From a2ffe39033d1f06efe947c3deecd1a416bd67697 Mon Sep 17 00:00:00 2001 From: Danila Fedorin Date: Thu, 5 Dec 2019 02:22:03 -0800 Subject: [PATCH] New draft of tech review for Kirsten. --- techreview.latex | 395 +++++++++++++++++++++++++++++++++-------------- 1 file changed, 278 insertions(+), 117 deletions(-) diff --git a/techreview.latex b/techreview.latex index 7f14e6f..775f671 100644 --- a/techreview.latex +++ b/techreview.latex @@ -1,4 +1,4 @@ -\documentclass[10pt, draftclsnofoot,onecolumn]{IEEEtran} +\documentclass[10pt, draftclsnofoot,onecolumn, compsoc]{IEEEtran} \def\changemargin#1#2{\list{}{\rightmargin#2\leftmargin#1}\item[]} \let\endchangemargin=\endlist @@ -30,6 +30,9 @@ \pagebreak +% TODO: She wanted something with "my role"? +% TODO: She wanted a more specific abstract. + \section{Role} My role in this project is the development of the smart application which will be used to control the Fenceless Grazing Collars in the field. This @@ -45,6 +48,35 @@ and electric stimuli. Additionally, the FGC aims to provide information about the behavior of the farm animals to their keepers, with the goal of improving their understanding of the livestock. +\section{Selection Criteria} +Because there is no ideal technical solution, the tradeoffs +of every possible technical solution must be analyzed through specific +critera. The Fenceless Grazing Collar system is intended for actual +use in the field by non-technical personnel, and therefore has +the following concrete design concerns: + +\begin{itemize} + \item \emph{Accessibility:} The system must be configurable and usable + without a significant learning curve. This means, among other things, + that the user interface of the system must be in some way familliar + to the users (farmers). + + \item \emph{Cost:} The entire idea of the project is to reduce the amount + of time and resources that is currently spent by farmers on herding + livestock. If the system is not cost-effective, it does not efficiently + replace farmers, and therefore, is not worth it to the users. + + \item \emph{Maintainability:} While this metric is not necessarily reflected + externally, it's nonetheless an important consideration. The software + written for this project must be easy to support and maintain in the future. + This is a goal for most long-term projects. + + \item \emph{Reliability:} The system must be reliable, since failures may lead + to loss of livestock or property. This means that communication between + the various components of the project must be effective and consistent, + and that bugs occurring in the codebase have a high cost to the users. +\end{itemize} + \section{Responsibilities} \subsection{Smart Device Application} A major component of this project is the creation @@ -53,15 +85,23 @@ the settings of the wireless collar devices. This section describes the components of this application. \subsubsection{Platform} -The Android platform makes up more than half of the market share -in the United States \cite{android-share}, with a majority of the remaining share taken -up by Apple iOS. Because iOS requires an Apple device to perform -development, and because Android devices are -on average more than 50\% cheaper than iPhone devices \cite{iphone-price}, we will -use the Android platform for developing the mobile application -associated with this project. -% https://www.businessinsider.com/android-stops-us-market-share-decline-2013-5 -% https://www.forbes.com/sites/amitchowdhry/2015/02/03/average-iphone-price-increases-to-687-and-android-decreases-to-254-says-report/#4d9d29a3539e +Either the Android or the iOS platform could be used +for the smart application. These platforms require entirely +different tooling and codebases, as well as different types +of devices (Apple phones only run iOS, while Android devices +only run Android). + +From the perspective of \emph{Accessibility}, Android is better +suited for this project. This is because the Android platform +makes up more than half of the market share in the United States \cite{android-share}, +with a majority of the remaining share taken up by Apple iOS. Furthermore, +because Android devices are on average 50\% cheaper than Apple devices\cite{iphone-price}, the barrier +to purchasing new technology to accomodate the smart application would +be lower if Android was used. This is also a benefit from the \emph{Cost} perspective. + +Because the Android platform has advantages in terms of both the \emph{Accessibility} +and the \emph{Cost} aspect, this is the optimal technology to use for the +FGC project. \subsubsection{Language} There exists a variety of available technologies and techniques suitable @@ -81,24 +121,29 @@ platform \cite{android-kotlin}. %https://facebook.github.io/react-native/ %https://www.theverge.com/2017/5/17/15654988/google-jet-brains-kotlin-programming-language-android-development-io-2017 -The use of JavaScript was ruled out due to its weak type system. -The language does not undergo a process called "type checking", -which means that certain bugs that can easily be detected -in other languages may go unnoticed in a JavaScript program -until they are observed in practice. Since this application -must reliably control the wireless collars, this is an -unnecessary risk. +The \emph{Maintainability} persepctive is most imprtant for decisions +in this area: The chosen language may significantly affect the difficulty +of developing and maintaining the smart application. From this perspective, +JavaScript-based frameworks such as Ionic and React Native are +at a disadvantage. They have weak type systems, which means that they +do not undergo a process called "type checking". This, in turn, +means that certain bugs that can easily be detected in other languages may +go unnoticed in a JavaScript program until they are observed in practice, making +it harder to ensure high software quality. Furthermore, the weak type system +property is also detrimental in terms of \emph{Reliability}: if bugs may +slip directly into the real-world usage of the application, there is +no way to guarantee consistent good behavior. -The choice then remains between Java and Kotlin, both -official languages of the Android platform. For our -purpose, Kotlin has advantages over Java. First, -Kotlin has a stronger type system, which, as described -above, helps prevents certain bugs before they are -observed in practice. Second, while Kotlin has access -to all features of the Android platform that Java does, -it has significantly less "boilerplate" code. This will -help reduce the amount of code requiring maintenance, -and thus reduce the number of mechanical issues with the final code. +Between Java and Kotlin, Kotlin has a still stronger type system, helping +prevent more bugs at compile-time than Java. It is also a language +with less "boilerplate" code, which eases maintenance and +reduces needless complexity in resulting software. Both +of these are benefits to \emph{Maintainability}. Besides this, +because Kotlin is a language that runs on the Java Virtual Machine, +there are no features of the Android platform that can be done +with Java and not with Kotlin. As such, Kotlin is at the very +least equally as powerful as Java, and with the benefits outlined +above, it should be the language of choice for the project. \subsubsection{Backwards Compatibility} Because vendors can customize the Android operating system prior @@ -127,53 +172,116 @@ attempt to leave the prescribed area. Each data point will be associated with the collar that produced it, such that behaviors of individual animals can be easily analyzed. -\subsubsection{Storage Type} -An SQL database will be used for storing the data generated -by the collars. The technology chosen for this is MariaDB. -MariaDB \cite{mariadb} was chosen because of its permissive license -and compatibility with MySQL \cite{mysql}, another SQL dialect that is very -commonly used in the industry. There exist other technologies -for storage, such as MongoDB, which do not use the relational -model used by SQL-based databases. The advantage of NoSQL databases -is the ability to store data in arbitrary formats, without -specifying prior schema. On the other hand, SQL-based -systems (including MySQL and MariaDB) support concurrency, -which is much more useful for our project due to the future -need to scale the fenceless grazing system. Additionally, -we already know the format of our data, making the benefit -of flexible data storage irrelevant to our use case. +\subsubsection{Storage Server} +Two major categories of storage servers are available at the +present time: SQL and NoSQL databases. SQL-based systems +store data in relational format, turning each piece of +data into a collection of related entries in one or more +tables. On the other hand, NoSQL systems store data in +arbitrary formats (i.e., as potentially more complex entities +than entries in a table). -% https://mariadb.org/ -% https://www.mysql.com/ +SQL and NoSQL are largely equivalent in terms of the criteria +presented at the top of this document. However, because +SQL systems enforce a particular format for data (in +contrast to NoSQL systems, which allow data of various, +unpredictable formats), and because the type of data +that will be stored by the FGC project is known ahead +of time, SQL systems offer a slight advantage to +\emph{Maintainability}. This is because "invalid" data +can be detected and reported immediately as it is stored in +the database. External applications that access the database +(such as the API server) may contain bugs pertaining to +data storage, and having an additional layer of verification +in the storage server will help detect and fix those bugs. + +Among SQL systems, three are most commonly used: MariaDB +\cite{mariadb}, MySQL\cite{mysql}, and SQLite. SQLite +is a simple SQL-based database system: it doesn't use +a standalone server, unlike MariaDB and MySQL, and +is simply used from inside other applications as a library. +MySQL and MariaDB, on the other hand, are server-based +database systems. Applications needing to store information +into the databases must first establish a connection to +the database server. MariaDB and MySQL are nearly identical +in terms of functionality, although MariaDB has a more +permissive license. + +In terms of \emph{Maintainability}, SQLite is at a slight +disadvantage to the rest of the SQL-based systems due to +its restricted feature set. SQLite supports a subset of +the operations supprted by MySQL and MariaDB, and therefore +may be restrictive in the long run. In terms of \emph{Cost}, +MySQL is at a disadvantage due to its prohibitive license: +it is non-free software in terms of price, and although +it has a "free" version, this version receives security +updates and fixes at a slower pace. Since MariaDB +is entirely free and open source, but maintains +the features of MySQL, it is best choice for this project. \subsubsection{Storage Hardware} -Because MariaDB and MySQL require a server machine, a Linux machine -will be used to host the database. This machine will be -a Raspberry Pi, a small and low-power Linux single-board -computer. This computer not only has a full-featured -Debian Linux operating system, but also allows for the -addition of hardware components, which will be used -for communicating with collars in the field. Among the -hardware components compatible with the Raspberry Pi \cite{raspi} is a -LoRa shield, which allows the Pi to send and receive -LoRa signals. This is ideal since the server machine will also act as the LoRa -gateway, serving as the liaison between the collars -deployed in the field and the rest of the project. +Because MariaDB and MySQL require a server machine, such +a machine must be part of the final design. This +can be achieved with a standard server computer, +a VPS host, or a single-board computer such as a Raspberry Pi \cite{raspi}. -Alternative implementations include an AVR-based microcontroller -equipped with ethernet and / or Bluetooth. Such a controller -would be integrated with a LoRa receiver and transmitter, -and programmed to send data to some other storage medium, -potentially a Virtual Private Server (VPS). This -implementation was dismissed due to the additional -complexity introduced by separating the LoRa receiver -and transmitter from the storage medium. This leaves -room for issues such as network errors between -LoRa and the SQL machine, which are entirely -avoided my using a Linux machine (Raspberry Pi) with support -for external hardware components (LoRa shield). +The Raspberry Pi is a promising option in terms of both +\emph{Cost} and \emph{Maintainability}. It is a small, single-board +computer that costs approximately \$30, and is fully +capable of running a Linux distribution such as +Debian (or Raspbian). After the initial cost +of the purchase, the Raspberry Pi will not consume +significant amounts of power, and thus remain very cheap +in the long run. The advantage to maintainability is +due to its official support for various additional +hardware modules, such as modules for the LoRa +protocol used in other parts of the project. By +combining the functionality of the LoRa receiver +and the API server, the complexity of the overall +system can be greatly reduced, making it easier +to design and maintain. -% https://www.raspberrypi.org/ +A VPS host would have both benefits and drawbacks +in terms of \emph{Maintainability} and \emph{Cost}, +although it is superior in terms of \emph{Reliability}. +A Virtual Private Server is a Linux server that is hosted +by another company for a monthly or yearly fee. The costs +of such servers are fairly small: common VPS packages +cost around \$5 per month, which is an adequate price +for such a product. However, unlike the Raspberry Pi, +these costs would accumulate over time, since a VPS +is a subscription-based service. Furthermore, unlike +the Raspberry Pi, the VPS does not support any additional +hardware. Therefore, another device would need to +be constructed to receive data from the Fenceless Grazing +Collars in the field, and forward that data to the VPS +for storage. This increases the complexity of the system, +thereby making software for it more difficult to develop and +maintain. The main advantage of a VPS host is that it +will be guaranteed to remain functional at all times +without the need for additional support, preventing +unexpected outages and therefore increasing reliability. + +For this project, a standard server machine is practically +the worst of both worlds. Unlike the Raspberry Pi, which +has support for additional hardware through a standardized +and officially supported interface, a regular server computer +would require additional modification to be able to interface +with LoRa devices in the field (or, just like the VPS host, +it would require an additional dedicated device to receive +and forward LoRa communication). Furthermore, unlike +a VPS host, it will potentially require support (in +events such as power outages or network failures), which +may result in temporary failures, a major downside from +the perspective of \emph{Reliability}. + +Between a VPS host and a Raspberry Pi, the latter appears +better suited for the project due the simplicity of integrating +it into the system. No additional hardware or components +would be necessary, reducing the surface area for failures +(thereby benefitting \emph{Reliability}), and the cost +after the initial investment would be minimal, benefitting \emph{Cost}. +As such, a Raspberry Pi should be used for this project. \subsection{App API Server} An API server is needed to allow users of the Android @@ -181,44 +289,93 @@ smart application to interact with the fenceless collars deployed on farm animals. Smartphones do not have hardware that can communicate using the LoRa protocol, which is used for collars. As such, an intermediate -device, a server that can communicate with both smartphones -and LoRa, must be used. +device - a server that can communicate with both smartphones +and LoRa - must be used. -\subsubsection{Server Technology} -The server will run on the aforementioned Linux machine. -This is because it is rather expensive, in terms of both -price and complexity, to introduce two server machines -for the individual tasks of SQL data storage and smartphone -app support. +\subsubsection{Server Hardware} +It is possible to use the aforementioned Linux storage machine +as the hardware for the API server. This would have benefits +in terms of \emph{Cost} and \emph{Maintainability}, +while having certain drawbacks in terms of \emph{Reliability}. +It would be cheaper to use the existing hardware for the API +server: no further purchases would be necessary besides +the Raspberry Pi, VPS, or other server described +in the sections above. Furthermore, such a configuration +would be maintainable, since it eliminates the complexity +of managing two separate server instances and their communication +with each other. On the other hand, this approach +would couple the health of the API server to the health +of the database server: if one suffers a hardware or critical +software failure, so will the other. This means that +the API/data server will be a major point of failure. -The server will use the Python \cite{python} programming -language, which is natively supported by the Raspberry Pi, -to host an HTTP(S) server on the local network. The -use of Python will allow for rapid development, and give -direct access to the server's LoRa hardware. While other -languages, such as Go, are also commonly used for API -implementations, they lack the first-class support that -Python receives from the Raspberry Pi project. Additionally, -Python is currently at the top of the popularity rankings -for programming languages, leading to increased access -to documentation and support. +The alternative approach would be to have a separate +API server. Since no special-purpose hardware would +be needed for this task, by the logic in the Storage Hardware +section, a VPS host would tbe most likely candidate for +such a server. This would bring in the advantage +of \emph{Reliability}, but could complicate the system +making it less \emph{Maintainable}. Furthermore, +since such a configuration would introduce the need +for communication between the API server and the data +server, this exposes more area for bugs and faults. -The Gunicorn \cite{gunicorn} server will be used to expose the Python -application to the Android application clients. While -other web servers, such as Apache and nginx, are very -commonly deployed in production, Gunicorn has excellent -support for Python, which will allow the team to -quickly develop the HTTP application. +Because it is cheaper and less complex to set up +the API server on the same hardware as the data +server (the Raspberry Pi), this should be the +configuration of choice for this project. -% https://www.python.org/ -% https://gunicorn.org/ +\subsubsection{Server Language} +Virtually any language can be used for developing an API. +Python \cite{python} and Rust are some of the viable +options for this project. + +Unlike Python, Rust a complicated type system, which is one of its +main selling points. The type system prevents many +classes of bugs that are common in software products, +especially those that use concurrency. It is therefore, +in a way, more \emph{Maintainable} than th other alternatives, +by the same logic that made Koltin more maintainable than +the alternatives for smart application development. + +Python is a dynamically typed language, and as such does +not benefit from type checking directly. It thus suffers +in comparison to Rust in terms of maintainability. However, +Python's main advantage for this project is that it is +the official language of the Raspberry Pi, and therefore +can be used for certain hardware manipulation tasks +that are more difficult in other languages. Because +the API server will be tied to LoRa, which is an external +hardware module, using Python for the entire +project will make the end product \emph{more} maintainable, +since there will be no need for additional frameworks, +libraries, or other components to bridge the gap between +the implementation language and the hardware. + +Because of its first-class support for hardware manipulation +on the Raspberry Pi, Python is significantly better for +this project than other possible languages. + +\subsubsection{Server Software} +Special software is typically used to expose a web application +to the outside world (which is necessary for other components, +such as the smart application, to access the API server). Apache, +nginx, and Gunicorn\cite{gunicorn} are valid options for this purpose. + +Gunicorn is superior in terms of \emph{Maintainability} due to +its simplicity. Whereas both Apache and nginx are general-purpose +server software, which have powerful configuration options and +complex execution models, Gunicorn does one thing: host Python +web applications. Since the API server is not intended to be +a complex system, it will not benefit from additional configurability +provided by Apache or nginx, and will therefore reduce the cognitive +load on team members as they work on the project. \subsubsection{Authentication} -The mobile app will use JSON web token (JWT) -to allow the mobile application to make multiple requests -without having to continuously provide a username -and password to the server. The idea of JWT is that -a JSON (JavaScript Object Notation) object, containing +JWT and Cookie-based approaches can be used to perform +authentication in the API server. + +The idea of JWT is that a JSON (JavaScript Object Notation) object, containing session information (such as the identity of the current user), is encrypted with information from the server. The encrypted version of this object is then sent to the user, and can be @@ -227,11 +384,14 @@ the token, the user cannot deliberately make changes to it, preventing security breaches. Once a user logs in, a JWT token will be generated, containing the identity of the user and an expiration date, and returned to the app. The app will then use the token for further -requests. +requests. This technology is useful in terms of \emph{Maintainablity}, +since it is widely supported by most programming languages, and does +not require the use of a browser. Thus, if the project migrates +to other platforms in the future, the technical cost of implementing +authentication on these new platforms will be negligible. -Cookie-based authentication systems were considered as an alternative; -This is commonly used in web applications, serving as a different -way to avoid authentication on every API or web request. However, +Cookie-based authentication systems are commonly used in web applications, +serving as a different way to avoid authentication on every API or web request. However, this approach will not generalize well. Using cookie or session-based authentication requires more extensive bookkeeping on the client side, which complicates the implementation of other clients. In the future, it is @@ -241,26 +401,27 @@ requiring the least additional implementation overhead is preferred. Because of this criterion, JWT-based authentication is better suited for the API server. +Because of the above issues with Cookie-based authentication, JWT +should be used for this project due to its benefits to \emph{Maintainability}. + \subsubsection{API} There are several ways of communicating data to the mobile application from the server. Because writing custom code to encode / decode data -send between the android application (client) and the server is +sent between the android application (client) and the server is time consuming and prone to errors, priority is given to existing encoding / decoding technologies. Of these technologies, Google's -ProtoBuf \cite{protobuf}, XML, and JSON are the most viable. JSON was chosen -for this project because of the ease with which it can be decoded, +ProtoBuf \cite{protobuf}, XML, and JSON are the most viable. + +% TODO I guess you can talk about protobuf need to regenerate shit + +JSON should be used for this project because of the ease with which it can be decoded, as well as due to its compatibility with the JavaScript ecosystem. Virtually every language (including Kotlin and Python) has support for JSON decoding that is well-tested and supported. This makes -JSON the most standard choice for an interchange format. - -The aforementioned HTTP server will thus provide a JSON API -to the client. During each HTTP requests, the client will -include a JWT token, and, once that token is verified, the server -will process the relevant information and return a JSON object -through HTTP containing the resulting data. - -% https://developers.google.com/protocol-buffers +JSON the most standard choice for an interchange format. These +benefits give a huge boost to \emph{Maintainability}, since +little additional work needs to be done to handle JSON in most +modern systems. \pagebreak \begin{thebibliography}{99}