Category Archives: Open Source

More Thimble bugs to work on (for Release 0.2)

The bugs I will be working on for my second Release (0.2) are both related to a previous issue I contributed to. I wrote a blog about it here.

The first one (Issue #1738) involves fixing the event’s callback parameters. The second (Issue #240) is to have the renamed project fully published and actually update on S3. (Amazon S3 is a simple storage service that Thimble stores its projects on).

I will continue with this post once I have finished working on these bugs.

Fixing bug in Thimble project using Persistence DevTools

This is a continuation to my previous blog on setting up Brackets, Thimble and choosing a bug to work on. Now that I have my local instance of Thimble installed, I can start working on the bug.

Bug to fix

The issue here is that renaming a project is not recognized as a change in an already published project, so the “Update published version” button is not available.

  • For example, an already published project named: ‘My Project’
  • thimble_savebutton

  • Rename it to ‘My Project renamed’ and click ‘Save’:
  • thimble_projectrename

  • Now if we click on the ‘Publish’ button, the ‘Update published version’ button is not available after renaming the project:
  • thimble_publishnobutton

Setting up workspace

In the previous post, I set up my local Thimble instance by Forking thimble.mozilla.org into my repo, then cloning the project from my repository. To start working on this bug, I created a new branch (lkisac_issue_719). This guide explains how to set up “Persistence with DevTools Workspaces”, so that I could edit the files on my local machine and have the changes applied server-side on a browser refresh. This is essentially done by mapping files/folders to your local computer.

Drag and drop your local project folder (thimble.mozilla.org) to the Source navigator pane in DevTools.

devtools_workspace

You can right click the filename in the Sources editor, then select ‘Reveal in navigator’ to make sure the file your working on is the one from your local folder.

Issues

Vagrant had to be reloaded each time in order to see the changes. This was fixed in this commit, so I pulled these latest changes to my repository.

I also had an issue with my vagrant commands returning this error:

There was an error while executing `VBoxManage`, a CLI used by Vagrant
for controlling VirtualBox. The command and stderr is shown below.
Command: ["showvminfo", "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00"]
Stderr:

Earlier while I was running ‘vagrant up’ I ran into the Windows BSOD (blue screen of death) which interrupted vagrant’s start up. After restarting my computer and running vagrant commands, I would get this error. I searched online for related issues but could not find a solution that worked (including doing a clean installation of vagrant and virtual box and following some suggestions from here). I finally came across this thread where the “.vagrant” file was mentioned. This is a metadata file that stores project info and must have been gotten corrupt when vagrant was interrupted. I tried moving this file outside of my project folder and was able to get vagrant up and running.

During development, one thing I ran into sometimes was after leaving thimble running in vagrant for a while then coming back to it, I would suddenly run into some 401 errors on my Thimble instance. A few ways I got around this was to:

  • Select ‘Disable cache’ under the Network tab in DevTools (this is a useful setting while doing web development for any cached data to not interfere while testing. It won’t affect your regular browser’s cache settings)
  • Sign out of Thimble account then sign back in

Code changes

One of the maintainers provided some guidance on which sections of code would need changing. There were 5 files I modified in the code.

publisher.js

  • Changed handleFileEvent to be on the Publisher prototype
  • Renamed handleFileEvent to showUnpublishedChangesPrompt

main.js

  • Moved ProjectRenameUtility.init call to bramble-ui-bridge.js
  • Added ProjectRenameUtility to BrambleEditor create options

bramble-ui-bridge.js

  • Now contains call to ProjectRenameUtility.init() after publisher init
  • init function receives csrfToken, appUrl and ProjectRenameUtility

bramble-editor.js

  • Pass csrfToken, appUrl and ProjectRenameUtility to BrambleUIBridge.init call.

project-rename.js

  • Call new showUnpublishedChangesPrompt from rename utility’s save function
  • Include publisher in the require, stored in var Publisher
  • Added publisher and bramble parameters in ProjectRenameUtility constructor and stored in this

Setting up breakpoints in DevTools was really useful for working through the code and testing functionality.

Example for setting a breakpoint at line 343

You can click on the line number in the online Sources editor to set the breakpoint.

devtools_breakpoints

Then step through the code with these commands (similar to Visual Studio):

  • F10 – step over
  • F11 – step in
  • Shift + F11 – step out of function

After much trial and error, I was finally able to see the ‘Update published version’ button after renaming the project.

thimble_publishbutton

Create Pull request

After I was able to get the expected results, I committed my changes to my lkisac_issue_719 branch and created my Pull Request.
When you commit code to open source projects, there will be one or more test builds that will likely need to be run to check if the code is valid with those tests. After pushing my changes to git, my first commit failed in the Travis CI build. It was noted that the bramble object in project-rename.js was undefined. So I fixed it in this commit, and the build passed successfully.
Additionally, the maintainer also provided some feedback so they could fully land this patch in Thimble. So I went back into my workspace, made the necessary changes, tested the functionality again, and committed the changes back to git.

Conclusion

It had been a while since I worked with much JavaScript code, so it took a little getting used to (prototypes, chaining, etc). Reading over mozilla’s documentation was helpful in understanding the function calling and how the objects were being passed so I could add my changes correctly.
The most difficult and time consuming part for me in this whole process was getting my workspace set up properly before even getting started on the code. Once I was finally set up and able to write and test code, it was a good experience working on the Thimble code and contributing to this open source project.

Git branching, merging & rebasing (Spoon-Knife repo example)

This post will go over some examples for branching, merging and rebasing using GitHub’s Spoon-Knife repository.

Fork & Clone repository

  1. Fork the Spoon-Knife repository from GitHub
  2. Clone repository to local computer:
    git clone https://github.com/lkisac/Spoon-Knife.git
    Confirm that we are on the master branch:
    git branch
  3. Create a new file called name.txt with first name in it:
    vi name.txt

    Len Isac
    
  4. Add and commit the name.txt file to Git:
    git add name.txt
    git commit -m "added name.txt file with first name"
  5. Confirm that you have 1 new commit on the master branch:
    git log

    commit d87e3b4dc4a9497d68fcbbfe3247409225d4a959
    Author: Len Isac Seneca <len_isac@hotmail.com>
    Date:   Sat Feb 4 10:48:55 2017 -0500
    
        added name.txt file with first name
    
  6. Since we would like to do this work on a new branch instead, we’ll create a new “name” branch, and merge it back into master later:
    git checkout -b name
    Confirm we are on newly created “name” branch:
    git branch
    Check to see that the master and name branch both point to the same commit:
    git show name
    git show master

    Since we created the “name” branch from inside the “master” branch, git will create the new branch pointing to master’s commit (“added name.txt file…”).
  7. Move the master branch back one commit:
    git checkout -B master HEAD~1
  8. Check that both branches are pointing to different commits now:
    git show name
    git show master

Merging

  1. Create third branch named “fav-food” from the master branch.
    git checkout -b fav-food
  2. Create food.txt file with a list of favourite foods
    vi food.txt

    Spaghetti
    Toast
    Beef curry
    
  3. Add and commit
    git add food.txt
    git commit -m "added food.txt - list of favourite foods"
  4. Create fourth branch named “name-and-food” from the name branch commit. Since we are inside the fav-food branch but want it to point to the “name” branch’s commit, we need specify it in the second parameter:
    git checkout -b name-and-food name
    Confirm new merge commit “Merge branch ‘fav-food’ into name-and-food”
    git log --max-count=1
    git ls

    We can also see now both the food.txt and name.txt files exist in this branch.

Merge conflicts

To demonstrate merge conflicts, 3 branches that contain the same “animals.txt” file will be created.

  1. Create “animals” test branch from master:
    git checkout -b animals master
  2. Now we’ll create an “animals.txt” file with 3 farm animals.
    vi animals.txt

    cow
    lamb
    horse
    
  3. Add & commit:
    git add animals.txt
    git commit -m "added animals.txt"
  4. Create a second “water-animals” branch pointing to the animals branch commit:
    git checkout -b water-animals
  5. Edit the “animals.txt” file and add 3 water animals:
  6. vi animals.txt

    cow
    lamb
    horse
    dolphin
    whale
    seahorse
    
  7. Add & commit:
    git add animals.txt
    git commit -m "added three water animals to animals.txt
  8. Create a third branch named “jungle-animals pointing to the animals branch commit:
    git checkout -b jungle-animals animals

  9. Edit the “animals.txt” file and add 3 jungle animals:
  10. vi animals.txt

    cow
    lamb
    horse
    tiger
    lion
    bear
    
  11. Add & commit:
    git add animals.txt
    git commit -m "added three jungle animals to animals.txt
  12. Switch to animals branch and merge water-animals into it:
    git checkout animals
    git merge water-animals

    animals.txt now contains the three water animals added in the water-animals branch.

    Updating b11d23a..3fce797
    Fast-forward
     animals.txt | 3 +++
     1 file changed, 3 insertions(+)
    

    Since the branch was ahead by 1 commit, git performs a fast-forward, which aligns the current branch with the branch being merged. git log shows its commit message, not the merge commit message this time.

  13. Now we’ll try merging the “jungle-animals” branch into animals
  14. git merge jungle-animals

    Auto-merging animals.txt
    CONFLICT (content): Merge conflict in animals.txt
    Automatic merge failed; fix conflicts and then commit the result.
    

    git status shows us that we have unmerged paths and need to fix conflicts, then run “commit”.

  15. Fix merge conflicts
  16. vi animals.txt

    cow
    lamb
    horse
    <<<<<<< HEAD
    dolphin
    whale
    seahorse
    =======
    tiger
    lion
    bear
    >>>>>>> jungle-animals
    

    HEAD (animals) and jungle-animals both use the same lines but have different contents. We have to choose which one we want or if we want both, then remove the conflict markers. If I simply remove the conflict markers, I can keep the contents from both branches making it six lines instead of three.

    cow
    lamb
    horse
    dolphin
    whale
    seahorse
    tiger
    lion
    bear
    
  17. Add the fixed file and commit the merge
  18. git add animals.txt
    git commit

    Merge branch 'jungle-animals' into animals
    
    Conflicts:
        animals.txt
    #
    # It looks like you may be committing a merge.
    # If this is not correct, please remove the file
    #   .git/MERGE_HEAD
    # and try again.
    
    
    # Please enter the commit message for your changes. Lines starting
    # with '#' will be ignored, and an empty message aborts the commit.
    # On branch animals
    # All conflicts fixed but you are still merging.
    #
    # Changes to be committed:
    #   modified:   animals.txt
    

    We can remove the “Conflicts: …” lines and commented lines, then save & exit.

    Merge branch 'jungle-animals' into animals
    

Rebasing, Squashing

Rebasing is a great tool to clean up your commits before being pushed to a remote repository. Although it can be done after pushing, it is not recommended unless you know for sure your commits have not been or not being used by any other developers.

We can combine a series of commits into one. This is called “squashing”.

git rebase -i master

pick b11d23a added animals.txt
pick 3fce797 added three water animals to animals.txt
pick 75e509d added three jungle animals to animals.txt
  • I want to combine all three commits into one, so I can squash the two later ones into the first b11d23a:
    pick b11d23a added animals.txt
    squash 3fce797 added three water animals to animals.txt
    squash 75e509d added three jungle animals to animals.txt
    

    Save and exit (Shift + ZZ)
    We see this error message:

    error: could not apply 75e509d... added three jungle animals to animals.txt
    
    When you have resolved this problem, run "git rebase --continue".
    If you prefer to skip this patch, run "git rebase --skip" instead.
    To check out the original branch and stop rebasing, run "git rebase --abort".
    
    Could not apply 75e509dbf9a48559f6ac5d6335554c240c72c45a... added three jungle animals to animals.txt
    

    Git replays each commit one-by-one on top of master and runs into the same conflict we had earlier. So we can fix the conflicts again by removing the conflict markers, then run:
    git add animals.txt

  • We are still running git rebase – since we have fixed the conflict we can continue with:
    git rebase --continue

     # This is a combination of 3 commits.
     # The first commit's message is:
     added animals.txt - three farm animals
    
     # This is the 2nd commit message:
    
     added three water animals to animals.txt
    
     # This is the 3rd commit message:
    
     added three jungle animals to animals.txt
    
     # Please enter the commit message for your changes. Lines starting
     # with '#' will be ignored, and an empty message aborts the commit.
     # rebase in progress; onto d0dd1f6
     # You are currently rebasing branch 'animals' on 'd0dd1f6'.
     #
     # Changes to be committed:
     #   new file:   animals.txt
    
  • We’ll leave the commit message as is with all three commit messages.

  • Save changes Shift + ZZ.
  • detached HEAD d3f629e] added animals.txt
     1 file changed, 9 insertions(+)
     create mode 100644 animals.txt
    Successfully rebased and updated refs/heads/animals.
    
  • Display rebase commit message
  • git log

    commit d3f629e2a2cad4bd61ced309dd15a06737d7ab4d
    Author: Len Isac Seneca <len_isac@hotmail.com>
    Date:   Sat Feb 4 13:39:29 2017 -0500
    
        added animals.txt
    
        added three water animals to animals.txt
    
        added three jungle animals to animals.txt
    

    Three last commits were successfully squashed into one commit.

    Setting up Vagrant, Brackets and Thimble and choosing a bug to work on

    Installing on Linux VM:

    My first attempt of installing Vagrant was on a Fedora 25 virtual machine.  I was able to get it running using the fedora/24-cloud-base box and changing a few config settings in the Vagrantfile (following suggestions from this issue), but wasn’t able to get it working within the thimble.mozilla.org project.  Even after changing the project’s Vagrantfile config settings to the one that I had working, keeping the project’s original config settings and installing some of the libvirt dependencies, ‘vagrant up’ would still fail. (EDIT: after leaving the VM for a couple hours then running vagrant up again, it suddenly worked.. still not sure how or why. There are several issues opened in GitHub regarding this, but leaving my computer for a few hours and coming back to it seems to have worked for now..)

    Installing Vagrant on Windows:

    Since I was having issues with the installation on my Linux VM, I decided instead to try the installation on my main OS running Windows 10 with Oracle’s Virtual box already installed.  I figured I could use my MinGW(linux-like terminal) with ConEmu already set up and run vagrant from here. Git bash can also be used which I also have included in ConEmu setup.  The installation process on Windows went smoothly, following the steps provided on the vagrantup website.

    Installing Brackets & Thimble:

    I continued to follow the setup instructions here, cloning both the Bramble and Thimble projects from GitHub and was able to run Vagrant successfully from the Thimble project directory.

    Here is a screenshot of Bramble up and running:

    bramble_windows_running

    and Thimble:

    thimble_running

    Thimble bug:

    From the list of good first bugs, I decided to work on a bug regarding the ‘last edited field’ in Issue #719.  I will follow up on this post with my progress working on this bug.

    Contributing to open source project – npm mysql packages.json

    I chose the npm mysql package to contribute to. In this post, I will go over the steps I took for contributing a minor fix to an open source project, using a Linux terminal, Vi editor and Git.

    Review Process

    Looking at the package.json file, I noticed that the repository field URL did not contain an absolute URL path.  So I ran the file through the package-json-validator:

    package-json_result

    The validator also recommends to include the “keywords” and “bugs” fields.  I read over the official npm documentation to see how to properly write these fields in package.json.

    Repository:

    "repository" :
      { "type" : "git",
        "url" : "https://github.com/npm/npm.git"
      }
    

    Keywords: An array of strings to help developers find the package using ‘npm search’.

    Bugs:

    {
      "url" : "https://github.com/owner/project/issues",
      "email" : "project@hostname.com"
    }
    

    I can adjust the fields now to match these formats.

    Adding my changes

    • Fork repository from https://github.com/mysqljs/mysql
    • Clone project to my local workspace: git clone git@github.com:lkisac/mysql.git
    • I used Vim editor to edit the package.json file: vi packages.json
    • Adjust repository URL to:
      "repository": {
        "type": "git",
        "url": "http://github.com/mysqljs/mysql"
      },
      
    • Add change to git staging: git add package.json
    • I thought it might be a good idea to keep this change separate from the “keywords + bugs” fields change for the sake of having separate commit messages for each, so I committed this change first: git commit -m "replaced repository url with valid repository url and type"

    • Then back to editing package.json in vi to add the “keywords” and “bugs” fields:
    "keywords": [
      "mysql",
      "sql",
      "database",
      "query"
    ],
    "bugs": "http://github.com/mysqljs/mysql/issues",
    

    Since I only included the issues URL in the bugs field, I followed the suggestion in npm of having it as only a single string instead of an object.

    • Add change: git add package.json
    • Commit change: git commit -m "added keywords + bugs url"
    • View changes in commits before pushing to repo: git show

    After running git show, I noticed the indentation was off (4 spaces instead of 2) on the “sql” keyword line.  I had already set my tabwidth setting to 2, but found out that this setting does not insert spaces in replacement of a tab character. To do this you have to set the shiftwidth and set expandtab. So I added these two lines to my ~/.vimrc file (as root user, or in Fedora /etc/vimrc):


    :set shiftwidth=2
    :set expandtab

    I switched back to my regular user and opened the package.json file again in vi.  If I press enter now from the “mysql” line, vi now automatically inserts 2 spaces at the beginning of the line.  To double check this, after hitting enter you can backspace and notice the cursor now moves back one space instead of the tabwidth’s set number of spaces.

    An alternate quick fix for this of course would be to use a regular expression substitution in vi on lines 13 to 23:

    :13,23s/\t/  /g

    This will replace all tab characters with two spaces from lines 13 to 23.

    Continuing with my changes… now I save these changes and exit vi CTRL + ZZ

    • See if changes are correct: git diff package.json
    • Spaces look good now, so I can add my changes: git add package.json
    • Commit changes to git: git commit -m "fixed vi indent to 2 spaces for sql keyword"

    Now looking at the last three commits I made, I figured since the latest commit was only a minor fix to the previous, I wanted to have the two as only one commit.

    git log --max-count 3

    git_last_3_commits_combine

    You can do this with a “git rebase”:

    git rebase --interactive HEAD~2

    This opens the git rebase file:

    git_rebase_start

    Here I want to pick the “added keywords + bugs url” as the main commit, and squash the “fixed vi indent…” commit into the main commit.

    git_rebase_squash

    Now I can hit Shift + ZZ to save my changes and exit git rebase.

    Git now displays the combination of commit messages:

    git_rebase_commit_msg

    Since I want to only use the “added keywords + bugs url” commit message, I can delete the 2nd commit message.

    git_rebase_new_commit_msg

    Hit Shift + ZZ again to save the changes and exit.

    Now the “fixed vi indent…” commit has been squashed with the “added keywords + bugs url” commit as one commit.

    git_final_squash_log

    Now I can push to the remote repository:
    git push

    One thing to note, as mentioned in github regarding git rebase, if the commits you squashed locally were previously already pushed to your remote repository, you would have to run:

    git push origin branch-name --force

    Although you have to be careful when choosing to do a rebase on already pushed commits and make sure they have not been reviewed or used in any way.  Use –force only only for very recent commits.

    Creating Pull Request

    Create the pull request by clicking the ‘Create pull request’ button in GitHub, then write a description of the changes you’ve made and create the new pull request.

    pull_request_checks

    Pull Request Review

    One of the collaborators were quick to respond within 3 minutes of the pull request.  It was noted that npm automatically populates these fields now so adjusting the packages.json file is not necessary.  The new npm also no longer uses the keywords array in the search and the validator URL is also valid as a shortcut URL for git.

    So I ended up creating a new issue for the validator-tool and the maintainer confirmed that the validator is slightly outdated and said they would be looking into updating it when they had the chance.

    cflow & cadvisor – reviewing successful patches for open source projects

    cflow

    cflow has one maintainer who is also the developer for the project.  It is still currently an active project which uses a mailing list for notifying of bug reports and suggestions.

    I had a look at the most recent patch from November 2016 regarding inverted trees missing the static function calls in the flow chart list.  The review process was seemingly quick and this method of review (mailing list, contrary to the workflow directly within GitHub) worked well in this case as the maintainer was able to review and apply the change successfully within a couple of weeks.  Although the workflow is not done directly on GitHub, the maintainer has made the project available from a Git repository as well as in CVS for contributors to work on the code.  The project’s official website also includes some well documented usage for Git.

    Using GIT at Savannah: http://savannah.gnu.org/maintenance/UsingGit/

    cadvisor

    For the cadvisor open source GitHub project, I had a look at the review process for the ‘Build & test cAdvisor with go 1.7.1 #1508’ pull request from October 19, 2016.  It addresses an issue that was opened August 22 of that year, regarding the release of cAdvisor v0.24 with go version 1.7.  Responses to the request were made the very same day by active participants in the project.  It was then reviewed by one of the maintainers on November 2, 2016, a few things were discussed regarding builds on a specific environment, and the pull request was finally merged on December 5, 2016.

    git_merge_tests

    Github is a great way for developers to share and contribute their ideas and suggestions to the open source community.  The review process for pull requests and merging those changes is safe because it requires all necessary tests to pass successfully before being able to merge them into the project.  I also found that although GitHub is the most used version control system for open source projects, the use of mailing lists (i.e. the cflow project), can work just as well if the scope of the project is not too large and there is only one or a few active maintainers.

    Docker – application container engine

    This is my first blog for OSD600 (Open Source Development) at Seneca.  As one of my first tasks for analyzing an open source project, I’ve chosen the docker project (https://github.com/docker) which currently has 100 contributors with just under 2000 open issues being worked on by the GitHub community.

    Docker is an application container engine that is hardware/platform independent, meaning it is capable of packing, shipping and running any application on any type of hardware or platform as a “lightweight” container (standardized unit); no particular language, framework, or packaging system is required.

    Containers are considered a better/faster distribution method compared to setting up a virtual machine, since the container does not require a full copy of the operating system, giving it a much faster startup time (minutes to seconds).  Here is an interesting thread on Quora (https://www.quora.com/What-is-the-difference-between-containerization-Docker-and-virtualization-VMWare-VirtualBox-Xen) detailing some of the advantages of containers over VMs and also simply going over some of the main differences between the two.

    Docker was released as open source in March of 2013.  The official website can be found here (https://www.docker.com).

    One of the major milestones for this project was on September 19, 2013 when Docker announced their major alliance with RedHat which made it possible for Fedora/RHEL compatibility, as well as Docker becoming a standard container within Red Hat OpenShift.  Another major open source project and current leading open source automation server Jenkins (https://jenkins.io/) has already had many plugins developed by the community dedicated to Docker compatibility (https://jenkins.io/solutions/docker/).