My Profile Photo

Notepad.ONGHU

Knowledge, without memory, is useless

• on | Ruby | Programming | RubyOnWindows | Gems | LBRS1 |

Ruby Learning by Reversing: Native Gems, Part 4

The first series of Learning by Reversing examines a Ruby native gem to understand how it works. Part 4 digs into the development Makefile and how it supports different activities during development.

Previously, in this series, we have had:

We ended the previous part with the question of Why are there two Makefiles? and moved on simply by saying that one is the development Makefile. This post gets into the details of the development Makefile so that you can see how it holds together and what you might need to do if you want to add more things.

The Development Makefile

Let’s look at what the Makefile looks like. The file is in some path somewhere and this is what is in it.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68

# Begin OS detection
ifeq ($(OS),Windows_NT) # is Windows_NT on XP, 2000, 7, Vista, 10...
    OPERATING_SYSTEM := Windows
    PATH_SEPARATOR := ;
else
    OPERATING_SYSTEM := $(shell uname)  # same as "uname -s"
	PATH_SEPARATOR := :
endif

# This gives debug output in the C code and some debugger flags, useful for... Debugging.
# See ext/fast_polylines/extconf.rb
DEBUG = # 1
EXT_NAME = fast_polylines
RUBY_FLAG = -Ilib -Iext -r $(EXT_NAME)
ALL_TARGETS = $(wildcard ext/$(EXT_NAME)/*.c) $(wildcard ext/$(EXT_NAME)/*.h)

##@ Utility

FMT_TITLE='\\033[7\;1m'
FMT_PRIMARY='\\033[36m'
FMT_END='\\033[0m'
.PHONY: help
help: ## Shows this help menu.
	@printf -- "                               FAST-POLYLINES\n"
	@printf -- "---------------------------------------------------------------------------\n"
	@awk ' \
			BEGIN {FS = ":.*##"; printf "Usage: make ${FMT_PRIMARY}<target>${FMT_END}\n"} \
	    	/^[a-zA-Z0-9_-]+:.*?##/ { printf "  $(FMT_PRIMARY)%-30s$(FMT_END) %s\n", $$1, $$2 } \
			/^##@/ { printf "\n$(FMT_TITLE) %s $(FMT_END)\n", substr($$0, 5) } \
		' $(MAKEFILE_LIST)

.PHONY: console
console: ext ## Runs an irb console with fast-polylines
	irb $(RUBY_FLAG)

.PHONY: test
test: ext ## Runs tests
	bundle exec rspec

.PHONY: rubocop
rubocop: ## Checks ruby syntax
	bundle exec rubocop

.PHONY: benchmark
benchmark: ext ## Run the benchmark
	bundle exec ruby $(RUBY_FLAG) ./perf/benchmark.rb

.PHONY: publish
publish: test ## Publish to rubygems
	gem build
	gem push fast-polylines-*.gem

ext/$(EXT_NAME)/Makefile: ext/$(EXT_NAME)/extconf.rb
	cd ext/$(EXT_NAME) && ruby extconf.rb --vendor

ext/$(EXT_NAME)/$(EXT_NAME).bundle: ext/$(EXT_NAME)/Makefile $(ALL_TARGETS)
	make -C ext/$(EXT_NAME)

##@ C extension

.PHONY: ext
ext: ext/$(EXT_NAME)/$(EXT_NAME).bundle ## Compiles the C extension

.PHONY: clean
clean: ## Cleans compiled stuff
	cd ext/$(EXT_NAME) && make clean
	rm ext/$(EXT_NAME)/Makefile
	rm fast-polylines-*.gem

This is our development Makefile and so, we expect it to help us during development. This means that we minimally want to be able to do the following without having to package and install the gem:

  • Rebuild the extension and see compiler warnings and errors
  • Test the extension
  • Interactively try to use the extension

In this post, we will not try to only read the Makefile and make sense of it but instead work forward from what we know we want to achieve.

Building the Native Extension

We have seen previously that the extension is built using the actual Makefile. That Makefile is generated during the gem installation by ext/extconf.rb. After that, the need is to just do make on that Makefile.

The picture below shows the link between the main parts related to the extension.

So, this is how it links up:

  1. ext (#1) is the main point and has a dependency on ext/$(EXT_NAME)/$(EXT_NAME).bundle (#2). This only requires the dependency to exist. Nothing else is needed to be done.
  2. This has a dependency on ext/$(EXT_NAME)/Makefile (#3) and $(ALL_TARGETS) (#4) – this has a rule attached to it which does make -C ext/$(EXT_NAME). If the dependency exists and is updated, it runs make on it. If not, the next item is run.
  3. The Makefile has a dependency on ext/$(EXT_NAME)/extconf.rb and has a rule to change directory and execute ruby extconf.rb --vendor. This ensures that the Makefile is generated by running extconf.rb during development.
  4. The last bit is $(ALL_TARGETS). This is specified further up in the file as ALL_TARGETS = $(wildcard ext/$(EXT_NAME)/*.c) $(wildcard ext/$(EXT_NAME)/*.h) – this sets all the C and H files as dependencies. If any of them change, the bundle needs to be updated, i.e., we need to run make.

The picture below shows the dependencies and how they feed in to building the extension during development. Any changes to the sources layer will result in the extension being rebuilt.

Testing the Native Extension

Once you build the native extension yourself, you can test it by doing bundle exec rspec that runs the RSpec tests for the native extension. This is what is reflected in the development Makefile as:

1
2
3

.PHONY: test
test: ext ## Runs tests
	bundle exec rspec

Running make test will ensure that this make target runs. This specifies ext as a dependency and will rebuild the extension if any of its dependencies have changed (as explained in the previous section).

Running rubocop

We now have a good expectation of how this would work. Once you build the native extension yourself, you would run rubocop by doing bundle exec rubocop. This is what is reflected in the development Makefile as:

1
2
3

.PHONY: rubocop
rubocop: ## Checks ruby syntax
	bundle exec rubocop

Running make rubocop will ensure that this make target runs. This again specifies ext as a dependency and that will rebuild the extension if any of its dependencies have changed.

Running the console

When you’re working with a new gem, it helps to have a console that requires the locally built gem so that you can try it out. For this, we want to run irb and get it to recognise lib and ext as being on the LOAD_PATH (by using the -I command line argument) so that it can find the locally built gem, and then doing a require by using the -r command line argument.

The actual command we need to do is:

irb -Ilib -Iext -r fast_polylines

We already store fast_polylines in a variable called EXT_NAME since we use it in a couple of other places also. We put -Ilib -Iext and the -r(EXT_NAME) into another variable called RUBY_FLAG so that it can be used in multiple places (you’ll see this again in the next section) and we end up with the below lines:

EXT_NAME = fast_polylines
RUBY_FLAG = -Ilib -Iext -r $(EXT_NAME)
irb $(RUBY_FLAG)

This is then exactly what is put for the make target called console as below. As before, ext is specified as a dependency so that it is rebuilt if anything changes.

.PHONY: console
console: ext ## Runs an irb console with fast-polylines
	irb $(RUBY_FLAG)

Running the benchmark

If you’re building a native gem for performance reasons, you will definitely want to add a benchmark to show that the new version is faster (and by how much). In the case of this gem, the performance benchmark is in a Ruby script ./perf/benchmark.rb and needs to be run. To run it manually on the command line, you would run Ruby while requiring the newly built native gem and the running the performance script.

As in the previous section, this means running ruby with the same RUBY_FLAG values: -Ilib -Iext -r(EXT_NAME) as we did for irb for the console command. The make target benchmark is defined as below with ext as a dependency as always.

.PHONY: benchmark
benchmark: ext ## Run the benchmark
	bundle exec ruby $(RUBY_FLAG) ./perf/benchmark.rb

Publishing the gem

If you have permission to push the gem and your environment is set up for it, you could use the gem push command to push the gem. The Makefile publish target does this, by setting test as a dependency (which in turn sets ext as a dependency), If the test target completes successfully, it will do gem build and gem push for the built gem.

.PHONY: publish
publish: test ## Publish to rubygems
	gem build
	gem push fast-polylines-*.gem

Cleaning up

Almost all makefiles will have a target for clean so that you can get a clean environment by doing make clean. This is done using:

.PHONY: clean
clean: ## Cleans compiled stuff
	cd ext/$(EXT_NAME) && make clean
	rm ext/$(EXT_NAME)/Makefile
	rm fast-polylines-*.gem

This does the following:

  • cd ext/$(EXT_NAME) && make clean => runs the make clean on the main Makefile
  • rm ext/$(EXT_NAME)/Makefile => remove the main Makefile
  • rm fast-polylines-*.gem => remove the previously built gem

Getting help

The Makefile has a help target that shows how to use the Makefile and that outputs formatted help text.

Two more things to understand

In the previous sections, we have gone through the structure of the development Makefile and hopefully, it is now clear and well understood. There are two more things that are helpful to understand.

What is .PHONY?

You might have noticed that each command started with .PHONY as in .PHONY test – what is .PHONY? This refers to a Makefile concept called Phony Targets. According to the make documentation, a phony target is one that is not really the name of a file. It is the name of a recipe to be executed when you make a specific request (e.g., make clean or make test or make rubocop, etc. in our case). If we did make rubocop without using a phony target and if there was a file called rubocop on the path, we might have a situation where make might decide to do nothing since the file has not changed. For this reason, we state .PHONY: rubocop which will result in make running the recipe irrespective of whether a file called rubocop exists since .PHONY is a special built-in target name.

The documentation also notes that prerequisites of .PHONY are always interpreted as literal target names, never as patterns (even if they contain ‘%’ characters). So, we must use the exact literals (e.g., rubocop, test, ext, clean, help, etc.) as the prerequisites for .PHONY.

Supporting Windows

A lot of Ruby developers work on Linux or Mac, and not at all on Windows. Although Ruby on Windows is now very good and most native extensions work very well and without any change, the issue often is a Makefile that excludes Windows support unintentionally.

In the case of this gem, there were a couple of related items that needed minor changes:

  1. Directory Separators
  2. Multiple arguments

We use -I to pass paths to irb and ruby as command line options. The description for -I says: “-I directory | Used to tell Ruby where to load the library scripts. Directory path will be added to the load-path variable (”$:“).”

There are two ways to provide multiple directories as command line options:

  1. You can use the -I option multiple times. This would be the same as doing -Ilib -Iext to pass the two directories. This method works on both Linux, Mac and also on Windows.
  2. The shortcut is to separate the multiple paths using a separator. On Linux/ Mac, the separator is : but on Windows, the separator needs to be ; – this means that -Ilib:ext does not work on Windows and -Ilib;ext does not work on Linux.

For this reason, it’s easiest if you use -Ilib -Iext so that there is no issue across platforms. However, if you do want to use the shortcut, you have to do some extra work to detect the OS and adjust the separator.

# Begin OS detection
ifeq ($(OS),Windows_NT) # is Windows_NT on XP, 2000, 7, Vista, 10...
    OPERATING_SYSTEM := Windows
    PATH_SEPARATOR := ;
else
    OPERATING_SYSTEM := $(shell uname)  # same as "uname -s"
	PATH_SEPARATOR := :
endif

After this, you can use PATH_SEPARATOR for RUBY_FLAG as below:

RUBY_FLAG = -I lib$(PATH_SEPARATOR)ext -r $(EXT_NAME)

However, I did find that because of the way the development Makefile is used in Windows, this method is not fool-proof since some parts of the process are run through bash and some using the Windows command processor.

For this reason, I recommend just using multiple -I options instead.

Summary

We have continued on our journey to stitch things together. In this post, we looked at the development Makefile and saw how it provide us targets for:

  • Building the native extension
  • Testing the native extension
  • Running rubocop
  • Running the console
  • Running the benchmark
  • Publishing the gem
  • Cleaning up
  • Getting help

We also looked at what Phony targets are in a Makefile, and also at how Windows is supported.

Looking ahead

This brings us to the end of Part 4 where we looked at the Makefile that holds everything together during development. Hopefully, you found it useful. Stay tuned for Part 5.

I will add links and references later, possibly in the last post of the series. If you have any comments, please feel free to leave them below.

comments powered by Disqus