Monday, March 27, 2017
Friday, March 17, 2017
Sonar in your build pipeline
If you're got a build pipeline in jenkins, sonar analysis should be one of your stages. But how to fail the build if sonar analysis fails?
Breaking the CI Build from the sonar docs is helpful, but here I go into some more detail on how to deal with this.
Step 1. Build your project in jenkins as the first step in your build pipeline
Step 2. Run the sonar analysis
This assumes you are using jacoco for coverage, and reading your front end results from an lcov report. The details of this are beyond the scope of this article.
Note that here I used mvn to run sonar. You can also run sonar with SonarRunner or other plugin.
Once sonar has run, in your jenkins console output you will see some INFO/DEBUG logs like this:
You can't access the project status yet, since the analysis might not be complete. You first need to check that the analysis is done. You can get the URL to get this info from the report-task.txt file that sonar write in your jenkins workspace (look at the debug info above)
In the jenkins shell, grep for the ceTaskUrl
This returns a JSON that looks something like this:
Grab the status with a sed (or use the amazing jq)
Wait for the analysis to complete
Once the loop completes, you can check the project status
This returns a JSON that looks like this:
So you can just check the projectStatus value and figure out if your jenkins build pipeline should fail or keep going.
For readability, here is the full code
Breaking the CI Build from the sonar docs is helpful, but here I go into some more detail on how to deal with this.
Step 1. Build your project in jenkins as the first step in your build pipeline
Step 2. Run the sonar analysis
mvn -e -X sonar:sonar \
-DskipTests \
-DXmx2g \
-DXX:MaxPermSize=1g \
-Dsonar.host.url=\${SONAR_HOST} \
-Dsonar.branch=\${BRANCH} \
-Dsonar.forceAnalysis=true \
-Dsonar.sourceEncoding=UTF-8 \
-Dmaven.test.failure.ignore=true \
-Dsonar.sources=. \
-Dsonar.junit.reportsPath=\${WORKSPACE}/**/target/surefire-reports \
-Dsonar.dynamicAnalysis=reuseReports \
-Dsonar.jacoco.reportPath=\${WORKSPACE}/target/jacoco.exec \
-Dsonar.javascript.lcov.reportPath=\${WORKSPACE}/frontend/target/coverage/js/lcov.info \
-Dsonar.java.binaries=\${WORKSPACE}/**/target/classes \
-Dsonar.verbose=true
This assumes you are using jacoco for coverage, and reading your front end results from an lcov report. The details of this are beyond the scope of this article.
Note that here I used mvn to run sonar. You can also run sonar with SonarRunner or other plugin.
Once sonar has run, in your jenkins console output you will see some INFO/DEBUG logs like this:
[INFO] 16:54:19.878 CPD calculation finished
[INFO] 16:54:20.300 Analysis report generated in 341ms, dir size=4 MB
[INFO] 16:54:20.835 Analysis reports compressed in 535ms, zip size=1 MB
[INFO] 16:54:20.835 Analysis report generated in /var/lib/jenkins/workspace/work_dev/target/sonar/batch-report
[DEBUG] 16:54:20.835 Upload report
[DEBUG] 16:54:20.988 POST 200 http://sonar.nymoenkumbera.no/api/ce/submit?projectKey=no.nymoenkumbera:work-project&projectName=fun&projectBranch=dev | time=153ms
[INFO] 16:54:20.990 Analysis report uploaded in 154ms
[INFO] 16:54:20.990 ANALYSIS SUCCESSFUL, you can browse http://sonar.nymoenkumbera.no/dashboard/index/no.nymoenkumbera:work-project:dev
[INFO] 16:54:20.990 Note that you will be able to access the updated dashboard once the server has processed the submitted analysis report
[INFO] 16:54:20.990 More about the report processing at http://sonar.nymoenkumbera.no/api/ce/task?id=AVrc-hOowKmvYVIu5IZf
[DEBUG] 16:54:20.997 Report metadata written to /var/lib/jenkins/workspace/fun_dev/target/sonar/report-task.txt
[DEBUG] 16:54:20.998 Post-jobs :
[DEBUG] 16:54:21.001 Execution getVersion
[DEBUG] 16:54:21.001 Execution stop
You can't access the project status yet, since the analysis might not be complete. You first need to check that the analysis is done. You can get the URL to get this info from the report-task.txt file that sonar write in your jenkins workspace (look at the debug info above)
In the jenkins shell, grep for the ceTaskUrl
declare -r SONAR_STATUS_URL=\$(less \${WORKSPACE}/target/sonar/report-task.txt | grep ceTaskUrl | sed -e 's/ceTaskUrl=//')
This returns a JSON that looks something like this:
{
"task": {
"id": "AVrc-hOowKmvYVIu5IZf",
"type": "REPORT",
"componentId": "AVSk2d1aVi-4YDCztopa",
"componentKey": "no.sb1.finance.bm:bm-master-project:dev",
"componentName": "Nettbank-bm dev",
"componentQualifier": "TRK",
"analysisId": "4870210",
"status": "SUCCESS",
"submittedAt": "2017-03-17T16:54:20+0100",
"startedAt": "2017-03-17T16:54:22+0100",
"executedAt": "2017-03-17T16:54:55+0100",
"executionTimeMs": 33030,
"logs": true
}
}
Grab the status with a sed (or use the amazing jq)
declare SONAR_STATUS=\$(curl -skg \${SONAR_STATUS_URL} | sed -e 's/.*status":"//' | sed -e 's/",.*//')
Wait for the analysis to complete
while ! [ "\${SONAR_STATUS}" == "SUCCESS" ] || [ "\${SONAR_STATUS}" == "CANCELED" ] || [ "\${SONAR_STATUS}" == "FAILED" ];
do
echo "Sonar analysis is: \${SONAR_STATUS}. Taking a nap while we wait..."
sleep 5
SONAR_STATUS=\$(curl -skg \${SONAR_STATUS_URL} | sed -e 's/.*status":"//' | sed -e 's/",.*//')
done
Once the loop completes, you can check the project status
declare -r SONAR_STATUS=\$(curl -skg \${SONAR_HOST}/api/qualitygates/project_status?projectKey=\${SONAR_PROJECT_KEY})
This returns a JSON that looks like this:
{
"projectStatus": {
"status": "OK",
"conditions": [
{
"status": "OK",
"metricKey": "new_coverage",
"comparator": "LT",
"periodIndex": 1,
"errorThreshold": "50",
"actualValue": "52.577319587628864"
},
{
"status": "OK",
"metricKey": "new_sqale_debt_ratio",
"comparator": "GT",
"periodIndex": 1,
"errorThreshold": "10",
"actualValue": "4.208754208754209"
},
{
"status": "OK",
"metricKey": "new_blocker_violations",
"comparator": "GT",
"periodIndex": 1,
"errorThreshold": "0",
"actualValue": "0"
},
{
"status": "OK",
"metricKey": "new_critical_violations",
"comparator": "GT",
"periodIndex": 1,
"errorThreshold": "0",
"actualValue": "0"
}
],
"periods": [
{
"index": 1,
"mode": "previous_version",
"date": "2017-03-15T11:49:15+0100",
"parameter": "2.25.0-SNAPSHOT"
},
{
"index": 2,
"mode": "previous_analysis",
"date": "2017-03-17T14:07:46+0100",
"parameter": "2017-03-17"
},
{
"index": 3,
"mode": "days",
"date": "2017-01-06T13:17:02+0100",
"parameter": "70"
}
]
}
}
So you can just check the projectStatus value and figure out if your jenkins build pipeline should fail or keep going.
For readability, here is the full code
#!/bin/bash
set -e
failOnSonarFailure() {
echo "Sonar analysis failed!"
# send a hipchat message to the right room
HIPCHAT_HOST="https://hipchat.nymoenkumbera.no"
HIPCHAT_ROOM="83"
HIPCHAT_AUTH="xxx"
echo "Sending a notification to room: \${HIPCHAT_ROOM}"
curl -skg --data "color=red&message_format=text&message=Oops! Sonar analysis of \${POM_ARTIFACTID} failed (boom) \${SONAR_HOST}/dashboard/index/\${SONAR_PROJECT_KEY}" \${HIPCHAT_HOST}/v2/room/\${HIPCHAT_ROOM}/notification?auth_token=\${HIPCHAT_AUTH}
exit 1
}
declare -r POM_GROUPID=\$(mvn org.apache.maven.plugins:maven-help-plugin:2.2:evaluate -Dexpression=project.groupId | egrep -v "(^\[INFO\]|^\[DEBUG\]|^\[WARNING\])")
declare -r POM_ARTIFACTID=\$(mvn org.apache.maven.plugins:maven-help-plugin:2.2:evaluate -Dexpression=project.artifactId | egrep -v "(^\[INFO\]|^\[DEBUG\]|^\[WARNING\])")
declare -r SONAR_PROJECT_KEY=\$(echo \${POM_GROUPID}:\${POM_ARTIFACTID}:\${BRANCH})
echo "Checking if analysis is finished.."
declare -r SONAR_STATUS_URL=\$(less \${WORKSPACE}/target/sonar/report-task.txt | grep ceTaskUrl | sed -e 's/ceTaskUrl=//')
declare SONAR_STATUS=\$(curl -skg \${SONAR_STATUS_URL} | sed -e 's/.*status":"//' | sed -e 's/",.*//')
while ! [ "\${SONAR_STATUS}" == "SUCCESS" ] || [ "\${SONAR_STATUS}" == "CANCELED" ] || [ "\${SONAR_STATUS}" == "FAILED" ];
do
echo "Sonar analysis is: \${SONAR_STATUS}. Taking a nap while we wait..."
sleep 5
SONAR_STATUS=\$(curl -skg \${SONAR_STATUS_URL} | sed -e 's/.*status":"//' | sed -e 's/",.*//')
done
echo "Sonar Task returned: \${SONAR_STATUS}"
if [ "\${SONAR_STATUS}" == "FAILED" ]; then
failOnSonarFailure
fi
echo "Checking status of analysis with sonar key: \${SONAR_PROJECT_KEY}"
declare -r SONAR_STATUS=\$(curl -skg \${SONAR_HOST}/api/qualitygates/project_status?projectKey=\${SONAR_PROJECT_KEY})
echo "Sonar WEB API returned:"
echo "\${SONAR_STATUS}"
if [[ \${SONAR_STATUS} == *'{"projectStatus":{"status":"ERROR",'* ]]; then
failOnSonarFailure
fi
Monday, November 7, 2016
Git simple merge conflict resolution
I was trying to merge something from feature/sonar_fixes to dev, but dev had changes from someone else. This is what I had to do to resolve conflicts:
Step 1: Fetch the changes (saving the target branch as FETCH_HEAD).
Step 2: Checkout the source branch and merge in the changes from the target branch. Resolve conflicts.
Thar be merge conflicts here: Open the files and look for FETCH_HEAD
Step 3: After the merge conflicts are resolved, stage the changes accordingly, commit the changes and push.
Step 1: Fetch the changes (saving the target branch as FETCH_HEAD).
git fetch origin dev
Step 2: Checkout the source branch and merge in the changes from the target branch. Resolve conflicts.
git checkout feature/sonar_fixes
git merge FETCH_HEAD
Thar be merge conflicts here: Open the files and look for FETCH_HEAD
Step 3: After the merge conflicts are resolved, stage the changes accordingly, commit the changes and push.
git add <file name with conflicts>
git commit
git push origin HEAD
Sunday, July 31, 2016
Datatables events not working on second page
I've revisited the datatables library for jquery. The new 1.10 version has a lot of improvements.
But I immediately ran into some trouble. After the easy zero config initialization I added a select box to one of the cells, and then added a a jquery "on change" event handler to that select box.
The problem is this works on the first page, but fails on subsequent pages, or if the page was ordered or filtered etc. This is specified in the "My events don't work on the second page" section of the Datatables FAQ and basically happens because Datatables manipulates the DOM.
My solution was to re-assign the event handler on each draw event, like this:
If I didn't specifically unbind the event handler (with the off), the event handler would get loaded loaded on each draw of the table. So I would start with 1 call, but then end up have 2, 3,4 calls each time I did a reorder, pagination search or so on.
Minutes after I fixed this issue, I found another nice slick way to handle this issue here: http://www.gyrocode.com/articles/jquery-datatables-why-click-event-handler-does-not-work/
which basically say you specify a selector as the second argument for the event handler, in this case a CSS class around the select box.
But I immediately ran into some trouble. After the easy zero config initialization I added a select box to one of the cells, and then added a a jquery "on change" event handler to that select box.
The problem is this works on the first page, but fails on subsequent pages, or if the page was ordered or filtered etc. This is specified in the "My events don't work on the second page" section of the Datatables FAQ and basically happens because Datatables manipulates the DOM.
My solution was to re-assign the event handler on each draw event, like this:
var table;
$(document).ready(function () {
table = $('#example')
.on('draw.dt', function () {
handleSelectEvent();
})
.DataTable();
});
var handleSelectEvent = function () {
console.log('in handleselect fn');
$('#example select').off('change')
.on('change', function () {
// do whatever
});
}
If I didn't specifically unbind the event handler (with the off), the event handler would get loaded loaded on each draw of the table. So I would start with 1 call, but then end up have 2, 3,4 calls each time I did a reorder, pagination search or so on.
Minutes after I fixed this issue, I found another nice slick way to handle this issue here: http://www.gyrocode.com/articles/jquery-datatables-why-click-event-handler-does-not-work/
which basically say you specify a selector as the second argument for the event handler, in this case a CSS class around the select box.
Wednesday, July 6, 2016
Change file permissions on git
If you push an executable file to git, its likely you wont have the right permissions.
As you can see its 644 (Ignore the 100)
Change it to 755:
flow.sh is now executable :)
D:\git\app>git ls-tree HEAD
100644 blob 390ce9106bc0e91a631cf345621b35b96bf74d20 flow.sh
As you can see its 644 (Ignore the 100)
Change it to 755:
D:\git\app>git update-index --chmod=+x flow.sh
D:\git\app>git status
On branch feature/app_move
Your branch is up-to-date with 'origin/feature/app_move'.
Changes to be committed:
(use "git reset HEAD ..." to unstage)
modified: flow.sh
D:\git\app>git commit -m "SLUG-11698 : Changing flow.sh file perms"
D:\git\app>git push
D:\git\app>git ls-tree HEAD
100755 blob 390ce9106bc0e91a631cf345621b35b96bf74d20 flow.sh
flow.sh is now executable :)
Thursday, June 23, 2016
Squash several commits in git into one
I had several commits on my branch, all related to the same change. Instead of merging all the commits into master, I wanted to squash all of them into one, and merge the single commit. Less junk on the master branch that way.
This https://gist.github.com/nicholashagen/2855167 has a shows a very nice way to merge and squash together. But since I use bitbucket(aka stash) and pull requests, the git merge --squash command doesn't work.
I had been looking for a slick way of squashing commits before generating a pull request, and recently found it.
I had 4 commits I wanted to squash (all the ones with the comment "Add sonar jobs for all apps") on the branch feature/add-sonar:
git log shows me the commit IDs
git rebase -i allows me to squash the ones from the current point (HEAD) back to the last n commits (in my case n = 4)
At this point I got an editor open up with the list of the last 4 commits, so I could do something with them:
Notice that the commits are in reverse chronological order - so the oldest one shows up first. I wanted to pick the oldest one, and squash the others. So I changed the text to look like this:
Ctrl-X (to save)
y (to confirm)
The rebase is successful:
Now all you have to do is push. But you need to force push.
Now git log shows only 1 commit:
Now you can merge that single commit in any way you choose :)
This https://gist.github.com/nicholashagen/2855167 has a shows a very nice way to merge and squash together. But since I use bitbucket(aka stash) and pull requests, the git merge --squash command doesn't work.
I had been looking for a slick way of squashing commits before generating a pull request, and recently found it.
I had 4 commits I wanted to squash (all the ones with the comment "Add sonar jobs for all apps") on the branch feature/add-sonar:
git log shows me the commit IDs
/git/docker-jenkins-master on feature/add-sonar * $ git log
commit 02041bddd50de5d5bd40bd3146ae44323fabb6a7
Author: Somaiah Kumbera <somaiah.kumbera@somewhere.com>
Date: Thu Jun 23 08:50:42 2016 +0200
Add sonar jobs for all apps
commit 17f11e416982bf486da2db732b3e1de580c59877
Author: Somaiah Kumbera <somaiah.kumbera@somewhere.com>
Date: Wed Jun 22 14:15:35 2016 +0200
Add sonar jobs for all apps
commit cc396d605bfe0db279d2beb11833880b3365edc7
Author: Somaiah Kumbera <somaiah.kumbera@somewhere.com>
Date: Wed Jun 22 13:07:43 2016 +0200
Add sonar jobs for all apps
commit affcae8432efef916d353dc4f204d4507422eebd
Author: Somaiah Kumbera <somaiah.kumbera@somewhere.com>
Date: Wed Jun 22 13:04:52 2016 +0200
Add sonar jobs for all apps
commit 29fa6593a0bc619d269c4b84b8f857b2dbe466e8
Merge: 78f0f8a 9751714
Author: Someone Else <someone.else@somewhere.com>
Date: Wed Jun 1 10:29:49 2016 +0200
Something unrelated
git rebase -i allows me to squash the ones from the current point (HEAD) back to the last n commits (in my case n = 4)
/git/docker-jenkins-master on feature/add-sonar * $ git rebase -i HEAD~4
At this point I got an editor open up with the list of the last 4 commits, so I could do something with them:
pick affcae8 Add sonar jobs for all apps
pick cc396d6 Add sonar jobs for all apps
pick 17f11e4 Add sonar jobs for all apps
pick 02041bd Add sonar jobs for all apps
Notice that the commits are in reverse chronological order - so the oldest one shows up first. I wanted to pick the oldest one, and squash the others. So I changed the text to look like this:
pick affcae8 Add sonar jobs for all apps
f cc396d6 Add sonar jobs for all apps
f 17f11e4 Add sonar jobs for all apps
f 02041bd Add sonar jobs for all apps
Ctrl-X (to save)
y (to confirm)
The rebase is successful:
[detached HEAD 4d51329] Add sonar jobs for all apps
2 files changed, 89 insertions(+), 1 deletion(-)
create mode 100644 path/dir/filename.xml
Successfully rebased and updated refs/heads/feature/add-sonar.
Now all you have to do is push. But you need to force push.
/git/docker-jenkins-master on feature/add-sonar * $ git push --force
Now git log shows only 1 commit:
/git/docker-jenkins-master on feature/add-sonar $ git log
commit 4d513295b564ad253c6d738496163c7d392af3d6
Author: Somaiah Kumbera <somaiah.kumbera@somewhere.com>
Date: Wed Jun 22 13:04:52 2016 +0200
Add sonar jobs for all apps
commit 29fa6593a0bc619d269c4b84b8f857b2dbe466e8
Merge: 78f0f8a 9751714
Author: Someone Else <someone.else@somewhere.com>
Date: Wed Jun 1 10:29:49 2016 +0200
Something unrelated
Now you can merge that single commit in any way you choose :)
Thursday, June 9, 2016
Sonar maven-surefire-plugin quirks
I was integrating my maven multi module project with sonar, and ran into some quirky behavior:
1. After some config changes I suddenly started seeing
Turns out I was calling mvn sonar:sonar with -Dsonar.exclusions and this exclusion was also set on the sonar server, under Administration > General Settings > Analysis Scope > Files as a source file exclusion.
Apparently excluding the same regex on the server and with the maven argument makes Sonar choke.
You can read the full explanation on my stackoverflow question
2. Some packages did not get code coverage with maven-surefire-plugin. Try as I might I always had 0% coverage on these packages. You can view my set up here. The maven output showed that the tests were run, and surefire did not produce any warnings.
It turned out that maven-surefire-plugin was included as a plugin in the build section in both the parent pom and the child pom. Hence the surefire plugin ran in the child module, but the resultant coverage, which was taken from the file specified by the sonar.jacoco.reportPath property did have these tests. Makes sense now of course, but it was a tricky bug to find :)
1. After some config changes I suddenly started seeing
Error updating database. Cause: java.sql.SQLIntegrityConstraintViolationException: ORA-00001: unique constraint (SONAR.PROJECTS_KEE) violated
The error may involve org.sonar.db.component.ComponentMapper.insert-Inline
The error occurred while setting parameters
Cause: java.sql.SQLIntegrityConstraintViolationException: ORA-00001: unique constraint (SONAR.PROJECTS_KEE) violated
Turns out I was calling mvn sonar:sonar with -Dsonar.exclusions and this exclusion was also set on the sonar server, under Administration > General Settings > Analysis Scope > Files as a source file exclusion.
Apparently excluding the same regex on the server and with the maven argument makes Sonar choke.
You can read the full explanation on my stackoverflow question
2. Some packages did not get code coverage with maven-surefire-plugin. Try as I might I always had 0% coverage on these packages. You can view my set up here. The maven output showed that the tests were run, and surefire did not produce any warnings.
It turned out that maven-surefire-plugin was included as a plugin in the build section in both the parent pom and the child pom. Hence the surefire plugin ran in the child module, but the resultant coverage, which was taken from the file specified by the sonar.jacoco.reportPath property did have these tests. Makes sense now of course, but it was a tricky bug to find :)
Subscribe to:
Posts (Atom)