04 Nov 2016

How we built Sentinel.la (II) – AngularJS and Netlify

“Apollo” is the codename of our UI. It was built with AngularJS because we wanted a single-page application (SPA). This is how it looks like:

AngularJS and Single-Page Apps.

An SPA ( Single-Page Apps) is a web application (or a website) that fits into a single page in order to give a more seamless experience to users, like a desktop application. Single Page Applications are becoming more popular for a good reason. SPAs can provide an experience that feels almost like a native app in the web, instead of sending a full page of markup, you can send a payload of data and then turn it into markup at the client side.

We decided to build an hybrid SPA. By hybrid I mean that, instead of treating the entire application as a single page application, we divide it into logical units of work or paths through the system. You end up with certain areas that result in a full page refresh, but the key interactions of a module take place in the same page without refreshing. For example, administration might be one “mini” SPA app while configuration is another.

Another advantage of AngularJS is that it uses the MVC approach, which is familiar to developers who have used Django or another MVC web development framework. Angular implements MVC by asking you to split your app into MVC components, then Angular manages those components for you and also serves as the pipeline that connects them.

Another advantage is that it puts the DOM manipulations where they belong. Traditionally, the view modifies the DOM to present data and manipulates the DOM (or invokes jQuery) to add behavior. With Angular, DOM manipulation code should be inside directives and not in the view. Angular takes the view just as another HTML page with placeholders for data.

Every SPA needs an API to consume. We built an API with Flask, that I’ll be reviewing in further posts,

Do not handle webservers, JAM Stack and Netlify.

Once we finished the UI and the API, we launched it over an nginx web servers farm. Also we decided to use a CDN to improve user experience in terms of speed and also to help us prevent site crashes in the event of traffic usage peaks. So, in this case, the CDN would help us to distribute bandwidth across multiple servers, instead of having to use just one server handling all the traffic.

While searching the internet looking for a good and affordable CDN we discovered Netlify and the JAMStack.

“JAM stands for JavaScript, APIs and Markup. It’s the fastest growing stack for building websites and apps: no more servers, host all your front-end on a CDN and use APIs for any moving parts[…] The JAMstack uses markup languages like HTML, CSS and Markdown to format and style our content, client-side Javascript to make it interactive and engaging and APIs to add persistence, real-time sync, real-world interactions, comments, shopping carts, and so on.“ – https://jamstack.org/

“Netlify is a unified platform that automates your code to create high-performant, easily maintainable sites and web apps” . – https://www.netlify.com/

So, we decided to use Netlify. It allow us to bring an automated platform to deploy our AngularJS app, because it follows the JAMStack, and also provide us with the best practices like CDN distribution, caching and continuous deployment with a single click – or command. That means not handling webservers, therefore less effort and work. For a startup like us that’s very valuable.

With Netlify we just push our site to their CDN because it pairs naturally with Git. That’s why you only need to pull, change and push the code to manage your site. After a git push, the newest version is immediately available everywhere and with their integrations you can set up any number of outbound webhooks so you can receive an e-mail or Slack notifications to notify for deploy information or build failures.

Netlify has a lot of features like DDoS protection, Snapshots, Versioning & Rollbacks, Instant Cache Invalidation, DNS Hosting, Domain Registration, etc. You can check more Netlify features here: https://www.netlify.com/features/

In the next post I’ll review the API and the AMQP topics about how we built our Software. Stay tuned.

31 Aug 2016

Job Opportunity! We are looking for a Software engineer: tools and infrastructure

Sentinelle Labs is a Latin American Startup with an innovative culture, for us the most important thing is to put us on the shirt of our customers while creating great things, taking care of the team and to be profitable. We’re a small and exponential team, therefore you won’t be just one more employee, you’re going to have the opportunity to leave your mark here. We love Linux and Open source and we also love home office. Our main efforts are focused at working with OpenStack and Ceph. We build SaaS, our stack is composed mainly of Python and Javascript and we deploy them on cloud technologies and containers platforms like Docker and Kubernetes.

In this position one of your main responsibilities will be to design and develop new features for https://www.sentinel.la our flagship product. You’ll also be responsible for deploying and supporting the platform with its components.

What are we looking for?

The first thing we want is a connection. We want to know about your dreams and to know if you are aligned with our values. If this happens, these are the requirements:

Being a person with self-learning ability.
Fluent English, ability to engage in conversation.
Experience as developer with a medium / high level both frontend and backend.
Experience with Python and JavaScript.
Medium / high knowledge level on Linux: Ubuntu / RedHat / CentOS
Experience in creation and consumption of RESTful APIs, knowledge on Git and MySQL / NoSQL administration.

What we offer?

The first thing is a job position that you’ll be passionate about using cutting-edge technologies while having the opportunity to make decisions to set the course of the platform, and for that as a compensation we offer:

Salary according to skills
Medical Insurance
Home Office
Phone Internet plan
Home Internet plan

If you are interested in working with us send your resume to guillermo@sentinel.la

09 Aug 2016

Sentinel.la now available at PyPI

Guillermo Alvarado
agent, devops, Logs, metrics, Monitoring, OpenStack, Roadmap, Sentinel.la
0 Comments

We are glad to announce that the sentinel.la agent is now available at Python Package Index (PyPI) the official third-party software repository for Python. https://pypi.python.org/pypi/sentinella

PyPI primarily hosts Python packages in the form of archives known as Python Eggs. Similarly to JAR archives in Java – Eggs are fundamentally ZIP files, but with the .egg extension, that contains the Python code for the package itself, and a setup.py file that holds the package’s metadata.

You can access to PyPI with several package managers, includings EasyInstall, PyPM and pip that use Pypi as the default source for packages and their dependencies.

So you will be able to install the sentinel.la agent with pip as following:

guillermo@xps13:~/$ pip install sentinella

With this, Sentinel.la is available for:

RedHat/CentOS through rpm package: https://github.com/Sentinel-la/sentinella-agent/releases
Ubuntu/Debian based systems through deb package : https://github.com/Sentinel-la/sentinella-agent/releases
SuSE Enterprise Linux/ OpenSuSE through python package: https://pypi.python.org/pypi/sentinella

Also, remember to vote for our presentation to OpenStack Summit at Barcelona:

Vote here: Double Win! Helping to consolidate OpenStack implementations (and build a Startup in the meantime)

Keep in touch with us while we’re building the next big thing,

Email: hello@sentinel.la

23 Apr 2016

Keep Austin Nerd! We’re in, are you? #OpenStackSummit

The Openstack Summit is attended by thousands of people from the whole world. This time the thirteenth release of OpenStack, codenamed Mitaka is ready. This version has improved user experience, manageability, and scalability. To see the whole agenda: https://www.openstack.org/summit/austin-2016/summit-schedule#day=2016-04-25

In the OpenStack Summit you will plan your cloud strategy and hear about market opportunities, latest products, tools and services like Sentinel.la, from the OpenStack ecosystem. We are ready to learn about operational experience directly from users.

So, our team will be attending the OpensStack Summit with this t-shirts, do you like it? what about getting one?

Please just send us a tweet to @The_sentinella to obtain a t-shiirt and a sticker and meet us at the event, we’ll be attending sessions and being around the Marketplace. See you at the Openstack Summit!

11 Jan 2016

Mastering the Openstack logs

how fast do you detect a problem in your deployment? no problem is so serious as it seems when talking about Openstack errors. The secret is in mastering the logs.Openstack and their components that run on top of it can generate all different types of messages, which are recorded in various log files.

Whenever a problem occurs in an Openstack deployment, the first place you should look up is in the logs. Analyzing the information provided in the logs, you may be able to detect what the problem is and where the error occurs. A lot of time the user interface only shows “An error occurred” but all the information regarding that error resides in the log file. You can use these messages for troubleshooting and monitoring system events.

Openstack has several services, and each of them have a log file, so there are a large number of log files. A good DevOps team that are managing an Openstack deployment -no matter the size- should need to locate the logs and learn how to work with them track the status and health of their deployment.

Where are the Openstack logs?

The Openstack services use a common location for their logs, in a default configuration of an openstack deployment, log files are located in subdirectories of /var/log directory:

Table from: http://docs.openstack.org/openstack-ops/content/logging_monitoring.html#openstack-log-locations

OpenStack uses the following logging levels: DEBUG, INFO, AUDIT, WARNING, ERROR, CRITICAL, and TRACE. What do each level mean?

Debug: Shows everything and is likely not suitable for normal production operation due to the sheer size of logs generated
Info: Usually indicates successful service start/stop, versions and such non-error related data. This should include largely positive units of work that are accomplished (such as starting a compute, creating a user, deleting a volume, etc.)
Audit: REMOVE – (all previous Audit messages should be put as INFO)
Warning: Indicates that there might be a systemic issue; potential predictive failure notice
Error: An error has occurred and an administrator should research the event
Critical: An error has occurred and the system might be unstable; immediately get administrator assistance

from: http://stackoverflow.com/questions/2031163/when-to-use-log-level-warn-vs-error/2031209#2031209

So the messages in the log files only appear if they are more “severe” than the log level that is set. For example using DEBUG we are allowing all log statements through. If you set DEBUG flag as FALSE only debug messages will be discarded. If you don’t want to see your logs polluted by INFO messages saying “hey, I’m here asking for somewhat!” then you can set VERBOSE as FALSE making WARNING the default.

*Those configurations are by service, so you need to change it on every conf file (of each service).

As you may know, there are logs by non-openstack components. OpenStack uses a lot of libraries, which do have their own definitions of logging. This logs can be wildly different because each one has their own definitions(MySQL, SQLAlchemy, KVM, OVS, Ceph,etc).

How do an Openstack log record looks?

The following is an example of a DEBUG log:

2016-01-04 22:41:36.297 DEBUG oslo_db.sqlalchemy.engines [req-af32b586-0aab-4846-b097-12604699d5ec None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py:256

Managing your logs

It is good practice to centralize events(logs) from our systems on a server, this server will collect the logs of our systems, classify them and then store them. So, there are two popular log collectors, Fluentd written in CRuby, used in Kubernetes and maintained by Treasure Data Inc and Logstash, written in JRuby and maintained by elastic.co. They have similar features. Both collectors have their own transport protocol, failure Detection and Fallback. Logstash uses Lumberjack protocol, and is Active-Standby only, in other hand Fluentd uses forward protocol and can be deployed as an Active-Active service (load balancing) or Active-Standby. You can read more about Logstash and Fuentd on their sites.

Whatever your decision is, you will need parse the Openstack logs to manipulate them. Yes, regex strikes back, welcome to the regex hell.

To save you a little time, we want to share the regular expression to parse the Openstack logs that Sentinel.la DevOps team wrote for such purposes:

Source: https://github.com/Sentinel-la/OpenstackRegexLog

OpenstackRegexLog

Regular expression to parse openstack logs

Example

1.- The following DEBUG log:

2016-01-04 22:41:36.297 DEBUG oslo_db.sqlalchemy.engines [req-af32b586-0aab-4846-b097-12604699d5ec None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py:256

Parsed and stored as json:

{
"time" : "2016-01-04 22:41:36.297",
"description" : "[req-af32b586-0aab-4846-b097-12604699d5ec None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION _check_effective_sql_mode /usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py:256",
"level" : "DEBUG",
"log_id" : null,
"component" : "oslo_db.sqlalchemy.engines ",
}

2.- The following WARNING log:

2016-01-04 22:41:35.221 19090 WARNING oslo_config.cfg [-] Option "username" from group "keystone_authtoken" is deprecated. Use option "user-name" from group "keystone_authtoken".