|
|
|
|
@@ -26,7 +26,7 @@ Hard to capture faults.
|
|
|
|
|
|
|
|
|
|
### Traditional error handeling
|
|
|
|
|
|
|
|
|
|
{% highlight c %}
|
|
|
|
|
{% highlight java %}
|
|
|
|
|
FILE *
|
|
|
|
|
openConfigFile(){
|
|
|
|
|
FILE * f = fopen("/path/to/config.conf");
|
|
|
|
|
@@ -63,9 +63,9 @@ This is some of the error handling real time programming have.
|
|
|
|
|
* Handling of unexpected errors
|
|
|
|
|
* More threads hanles errors
|
|
|
|
|
* Can not test the conventional way
|
|
|
|
|
* Can only show extistence of errors
|
|
|
|
|
* Can not find errors in specification
|
|
|
|
|
* Can not find race conditions
|
|
|
|
|
* Can only show extistence of errors
|
|
|
|
|
* Can not find errors in specification
|
|
|
|
|
* Can not find race conditions
|
|
|
|
|
|
|
|
|
|
The fault path is shown under.
|
|
|
|
|
|
|
|
|
|
@@ -103,8 +103,8 @@ To test how the systems responds for a unknown error is to insert a failed accep
|
|
|
|
|
**Dynammic redunancy**
|
|
|
|
|
|
|
|
|
|
* Relies on detecting the error and recovering
|
|
|
|
|
* Resend if timeout and not receiving "ack"
|
|
|
|
|
* Go with default if no messages have been received
|
|
|
|
|
* Resend if timeout and not receiving "ack"
|
|
|
|
|
* Go with default if no messages have been received
|
|
|
|
|
* The acceptancetest must be good.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@@ -128,29 +128,29 @@ Find the failure modes: What could go wrong?
|
|
|
|
|
**Step 3: Handling with redundancy**
|
|
|
|
|
|
|
|
|
|
* Have multiple copies of the the information
|
|
|
|
|
* Use only the newest
|
|
|
|
|
* Use only the newest
|
|
|
|
|
|
|
|
|
|
#### Example with communication function
|
|
|
|
|
|
|
|
|
|
**Step 1: Failure modes**
|
|
|
|
|
|
|
|
|
|
* Message
|
|
|
|
|
* Lost
|
|
|
|
|
* Delayed
|
|
|
|
|
* Corrupted
|
|
|
|
|
* Duplicated
|
|
|
|
|
* Wrong recipient
|
|
|
|
|
* Lost
|
|
|
|
|
* Delayed
|
|
|
|
|
* Corrupted
|
|
|
|
|
* Duplicated
|
|
|
|
|
* Wrong recipient
|
|
|
|
|
|
|
|
|
|
**Step 2: Detection, Merging of errormodes and error injection**
|
|
|
|
|
|
|
|
|
|
* Adding information to message
|
|
|
|
|
* Checksum
|
|
|
|
|
* Session ID
|
|
|
|
|
* Sequence number
|
|
|
|
|
* Checksum
|
|
|
|
|
* Session ID
|
|
|
|
|
* Sequence number
|
|
|
|
|
* Adding "ack" on well recieved messages
|
|
|
|
|
* All errors will be treaded as "Lost message"
|
|
|
|
|
* Injection
|
|
|
|
|
* Occasionally throw away some messages
|
|
|
|
|
* Occasionally throw away some messages
|
|
|
|
|
|
|
|
|
|
**Step 3: Handling with redundancy**
|
|
|
|
|
|
|
|
|
|
@@ -179,7 +179,7 @@ There are three solutions:
|
|
|
|
|
* Store a checkpoint
|
|
|
|
|
* Do the "side effects"
|
|
|
|
|
2. Process pairs
|
|
|
|
|
* Crash and let an another process take over
|
|
|
|
|
* Crash and let an another process take over
|
|
|
|
|
3. Presistent processes
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@@ -198,3 +198,343 @@ A transaction is a design framework for Damage Confinement and Error Recovery.
|
|
|
|
|
* **C**oncistency: Leaves the system in a consistent state when finished
|
|
|
|
|
* **I**solation: Errors does not spread
|
|
|
|
|
* **D**urability: Results are not lost
|
|
|
|
|
|
|
|
|
|
### Atomic Actions
|
|
|
|
|
|
|
|
|
|
**Resumption vs. Termination mode**
|
|
|
|
|
* If we continue where we were (e.g. after the interrupt) --> *Resumption*
|
|
|
|
|
* If we continue somewhere else (i.e. terminating what we where doing) --> Termination
|
|
|
|
|
|
|
|
|
|
**Async Notification (AN) = Low level thread interaction**
|
|
|
|
|
* Async event handling. ("Signals") (resumption)
|
|
|
|
|
* Modeled after a HW interrupt
|
|
|
|
|
* Can be sent to the correct thread
|
|
|
|
|
* Can be handled, ignored, blocked --> The domain can be controlled.
|
|
|
|
|
* Often lead to polling
|
|
|
|
|
* Could rather skip the signal and poll a status variable or a message queue
|
|
|
|
|
* Useless
|
|
|
|
|
* ATC --> Async transfer of Control (termination)
|
|
|
|
|
* Canceling threads
|
|
|
|
|
* setjmpt/longjmp could convert signals to ATC (not really, but still)
|
|
|
|
|
* ADA: a strictured mechanism for ATV is integraded with the selected statement
|
|
|
|
|
* RT Java: A structured mechanism for ATC is integraded with the exception-handling mechanism
|
|
|
|
|
|
|
|
|
|
#### Cancelling threads
|
|
|
|
|
|
|
|
|
|
**Yes, killing threads is ATC!**
|
|
|
|
|
|
|
|
|
|
* Can make termination model by letting domain be a thread
|
|
|
|
|
* "Create a `doWork` thread, and kill it if the action fails"
|
|
|
|
|
* Ca still control domain by disabling "cancelstate"
|
|
|
|
|
|
|
|
|
|
**But, but, but: It leaves ut in undifined state!?**
|
|
|
|
|
* Not if we have...
|
|
|
|
|
* Full control over changed state (like logs or recovery points) or some other way of recovering well.
|
|
|
|
|
* A lock manager that can unlock on behalf of killed thread
|
|
|
|
|
* Some control of where we were killed (like nok in the middle of a lock manager or log call)
|
|
|
|
|
* An this is what we have!
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## Shared variable synchronization
|
|
|
|
|
|
|
|
|
|
### Non-Preemptive scheduling
|
|
|
|
|
|
|
|
|
|
Controlling a pump filling a tank.
|
|
|
|
|
|
|
|
|
|
**Spec:**
|
|
|
|
|
* Every second: measure the water level of the tank and generate the reference to the pump
|
|
|
|
|
* 10 times a second: Set the power of the pump motor
|
|
|
|
|
* Do some GUI: let the human control the process
|
|
|
|
|
|
|
|
|
|
#### A trivial solution: "Cyclic Exectutive"
|
|
|
|
|
|
|
|
|
|
{% highlight java %}
|
|
|
|
|
oldTime = now();
|
|
|
|
|
i = 0;
|
|
|
|
|
while(true) {
|
|
|
|
|
i = i + 1;
|
|
|
|
|
if (i % 10 == 0) {
|
|
|
|
|
i = 0;
|
|
|
|
|
calculatePumpReference();
|
|
|
|
|
}
|
|
|
|
|
controlPump();
|
|
|
|
|
do {
|
|
|
|
|
handleUserEvent();
|
|
|
|
|
} while(now() < oldTime + 0.1);
|
|
|
|
|
oldTime = oldTime + 0.1;
|
|
|
|
|
}
|
|
|
|
|
{% endhighlight %}
|
|
|
|
|
|
|
|
|
|
**Drawbacks**
|
|
|
|
|
|
|
|
|
|
* OK tasks?
|
|
|
|
|
* Timing hard to tune (what if pump sampling should be $\pi$/10?)
|
|
|
|
|
* Overload (what if `calucaltePumpReference` uses more than 1/10 seconds?)
|
|
|
|
|
* How to add new tasks? (Everything is coupled)
|
|
|
|
|
* Waste of time in the do-loop?
|
|
|
|
|
* What is priority of `handleUserEvents`?
|
|
|
|
|
* How are erros, exceptions, alarms etc. handled?
|
|
|
|
|
|
|
|
|
|
#### Better soulution with Non-preemptive scheduler
|
|
|
|
|
|
|
|
|
|
* *3 taskts* administered by a scheduler
|
|
|
|
|
* The scheduler takes care of who runs and timing
|
|
|
|
|
* Scheduler often inculuded in OSes
|
|
|
|
|
* Introducing priorities
|
|
|
|
|
|
|
|
|
|
{% highlight java %}
|
|
|
|
|
/**
|
|
|
|
|
* scheduler_registerThread(function, time, priority)
|
|
|
|
|
* Higher priority numer means higher priority in scheduler
|
|
|
|
|
*/
|
|
|
|
|
main() {
|
|
|
|
|
scheduler_registrerThread(controlPump, 0.1, 3);
|
|
|
|
|
scheduler_registrerThread(calculatePumpReference, 1, 2);
|
|
|
|
|
scheduler_registrerThread(handleUserEvents, 0.2, 1);
|
|
|
|
|
scheduler_mainLoop();
|
|
|
|
|
}
|
|
|
|
|
{% endhighlight %}
|
|
|
|
|
|
|
|
|
|
**Some notes on priorities**
|
|
|
|
|
* Priority is generally not important; rather, the main rule is to give higher priority to shorter-deadline tasks.
|
|
|
|
|
* This allows tasks to reach its deadlines.
|
|
|
|
|
* ... but this is not always the case - if e.g. the tasks are cooperating
|
|
|
|
|
* We still handle overload badly
|
|
|
|
|
* And: What connection between deadline and priority to start with?
|
|
|
|
|
* Is this a good dependency seen from a code quality perspective?
|
|
|
|
|
|
|
|
|
|
### Pros and cons of nonpreemptive scheduling
|
|
|
|
|
|
|
|
|
|
| **Pros** | **Cons** |
|
|
|
|
|
| :--------------------------------------------- | :------------------------------------------------------------------------- |
|
|
|
|
|
| Simple, intuitive, predictable | C macro hell |
|
|
|
|
|
| No kernel | Threads must cooperate <-- a form of dependency breaking module boundaries |
|
|
|
|
|
| Fast switching times | Heavy threads must be divided |
|
|
|
|
|
| Some elegant sunchronization patterns possible | Can we handle blocking of library functions? |
|
|
|
|
|
| | Unrobust to errors |
|
|
|
|
|
| | Unrobust to (heavy) error handling |
|
|
|
|
|
| | Hard to tune at end of project |
|
|
|
|
|
{: .table-responsive-lg .table }
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### Preemptive Kernel
|
|
|
|
|
|
|
|
|
|
* Preemption, thread objects and the timer interrupt
|
|
|
|
|
* Enabling synchronization: Busy waiting, tes-and-set, disabling the timer interrupt
|
|
|
|
|
* Blocking and suspend & resume
|
|
|
|
|
* An API for synchronization? Semaphores!
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
#### Preemption
|
|
|
|
|
|
|
|
|
|
* Make a handler for a timer interrupt
|
|
|
|
|
* Store all registers (including IP & SP) in a "thread object"
|
|
|
|
|
* Organize queue of processes (Round Robin e.g. - a collection of thread objects?)
|
|
|
|
|
* Can synchronize by: `while(!ready);` (busy wating, "spin locks")
|
|
|
|
|
|
|
|
|
|
**Bad solution**
|
|
|
|
|
|
|
|
|
|
{% highlight java%}
|
|
|
|
|
while(lock==1) {}
|
|
|
|
|
lock = 1;
|
|
|
|
|
// We may run
|
|
|
|
|
lock = 0;
|
|
|
|
|
{% endhighlight %}
|
|
|
|
|
|
|
|
|
|
**Better solution**
|
|
|
|
|
|
|
|
|
|
{% highlight java%}
|
|
|
|
|
void t1() {
|
|
|
|
|
flag1 = 1; // Declare my intention
|
|
|
|
|
turn = 2; // But try to be polite
|
|
|
|
|
while(flag2 == 1 && turn == 2) {}
|
|
|
|
|
// We may run
|
|
|
|
|
flag1 = 0;
|
|
|
|
|
}
|
|
|
|
|
{% endhighlight %}
|
|
|
|
|
|
|
|
|
|
##### Looking more closely at the arsenal
|
|
|
|
|
|
|
|
|
|
**How can we make basic synchronization under preemption?**
|
|
|
|
|
|
|
|
|
|
* Spin locks (wasting time and cpu)
|
|
|
|
|
* Test&Set (swap) assembly instruction (atomic, but not obvious)
|
|
|
|
|
* Disable interrupt (steals control from OS/scheduler)
|
|
|
|
|
|
|
|
|
|
**But**
|
|
|
|
|
* If we disable the timer interrupt we don not have preemption any more
|
|
|
|
|
* And... Are these good abstractions in the application programmer domain?
|
|
|
|
|
|
|
|
|
|
#### Blocked threads
|
|
|
|
|
|
|
|
|
|
**Let us introduce another queue; the collection of threads not running, waiting for something**
|
|
|
|
|
|
|
|
|
|
* Fixes the bad performance of spin locks. Is conceptually better.
|
|
|
|
|
* `suspend` moves a thread object from "run" queue to "blocked" queue
|
|
|
|
|
* `resume` moves it back.
|
|
|
|
|
|
|
|
|
|
##### Two bad solutions
|
|
|
|
|
|
|
|
|
|
{% highlight java%}
|
|
|
|
|
t1(){
|
|
|
|
|
while(busy == 1) suspend();
|
|
|
|
|
busy = 1; // It is free; tak it - No
|
|
|
|
|
// Run
|
|
|
|
|
busy = 0; // Release resource
|
|
|
|
|
|
|
|
|
|
resume t2 // No
|
|
|
|
|
}
|
|
|
|
|
{% endhighlight %}
|
|
|
|
|
|
|
|
|
|
or
|
|
|
|
|
|
|
|
|
|
{% highlight java%}
|
|
|
|
|
t1(){
|
|
|
|
|
while(TestNSet(busy, 1) == 1) suspend();
|
|
|
|
|
// We own resource
|
|
|
|
|
// Run
|
|
|
|
|
busy = 0;
|
|
|
|
|
|
|
|
|
|
resume t2 // No
|
|
|
|
|
}
|
|
|
|
|
{% endhighlight %}
|
|
|
|
|
|
|
|
|
|
##### The suspend/resume problem
|
|
|
|
|
|
|
|
|
|
{% highlight java%}
|
|
|
|
|
// Global variables
|
|
|
|
|
bool g_initDone = False;
|
|
|
|
|
|
|
|
|
|
// Threads
|
|
|
|
|
t1(){ t2(){
|
|
|
|
|
/* Do init */ if (g_initDone == False) {
|
|
|
|
|
g_initDone = True; Suspend();
|
|
|
|
|
resume(t2) }
|
|
|
|
|
// Continue executing // Continue exectuting
|
|
|
|
|
} }
|
|
|
|
|
{% endhighlight %}
|
|
|
|
|
|
|
|
|
|
#### Priorities
|
|
|
|
|
|
|
|
|
|
* Threads mey have different *priorities*. (A sortet run-queue, or more of them.)
|
|
|
|
|
* Only if there are no running threads on a higher priority, a thread will run.
|
|
|
|
|
* We are not aiming for some sens of fairness (!). But predictability.
|
|
|
|
|
* And priorities supports schedulability proofs.
|
|
|
|
|
* But we open ourselves up to *starvation*. A thread may not ever get to run, even if it is runnable.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
#### Application-level syncronization
|
|
|
|
|
|
|
|
|
|
**SO, the application programmer needs some syncronozation primitives...**
|
|
|
|
|
|
|
|
|
|
* `sleep()`? - Ok
|
|
|
|
|
* Publish `suspend` and `resume` - No
|
|
|
|
|
* Events (`wait` and `signal`) - Just named versions of suspend & resume semantics.
|
|
|
|
|
* Fixes the need to know aboud "thread objects". But no
|
|
|
|
|
* ...or "Condition variables" - same
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### Semaphores
|
|
|
|
|
|
|
|
|
|
**A counting semaphore**
|
|
|
|
|
|
|
|
|
|
* `signal(SEM)` increases the counter (possibly resuming a thread waiting for the semaphore)
|
|
|
|
|
* `wait(SEM)` decrements the counter - will block (be suspended) `if SEM == 0`
|
|
|
|
|
* The semaphores value can not be negative
|
|
|
|
|
* Of course; These calls are protected from interleaving by disabling the timer interrupt
|
|
|
|
|
|
|
|
|
|
**We solve beautifully:**
|
|
|
|
|
* Mutual Exclusion
|
|
|
|
|
* Conditional Synchronization (ref `suspend`/`resume`)
|
|
|
|
|
* Basic resource allocation
|
|
|
|
|
|
|
|
|
|
**Semaphore variations**
|
|
|
|
|
|
|
|
|
|
* `wait` and `signal` nay take parameter value to add or subtract
|
|
|
|
|
* `getValue(SEM)` returning the value of the semaphore. (Fishy)
|
|
|
|
|
* BInary semaphores (`signal` will fail `if SEM == 1`)
|
|
|
|
|
* Who is woken at `signal` (FIFO, Arbitrary, Highest priority)
|
|
|
|
|
* The mutex
|
|
|
|
|
* binary
|
|
|
|
|
* ownership
|
|
|
|
|
* allows mulitple waits by owner
|
|
|
|
|
* regions (may be released by Javas `wait` or POSIX condition variables)
|
|
|
|
|
* RTFM
|
|
|
|
|
|
|
|
|
|
**Semaphore challenges**
|
|
|
|
|
|
|
|
|
|
* Breaks modules (both ways)
|
|
|
|
|
* Does not scale!
|
|
|
|
|
* Deadlocks
|
|
|
|
|
* Global analysis --> Does not scale
|
|
|
|
|
* Can not release "temporarily
|
|
|
|
|
* "Limited expressive power". Some reasonalbe problems are hard to solve
|
|
|
|
|
* Ref ["The Little Book of Semaphores"](https://greenteapress.com/semaphores/LittleBookOfSemaphores.pdf)
|
|
|
|
|
|
|
|
|
|
### Why shared-variable synchronization
|
|
|
|
|
|
|
|
|
|
**Why not?**
|
|
|
|
|
|
|
|
|
|
* "Shared variables" is bad code quality
|
|
|
|
|
* Ref global variables, and data members in module interfaces
|
|
|
|
|
* An obvious bottleneck? Scales terribly
|
|
|
|
|
* "Variables" are passive objects
|
|
|
|
|
* They can not protect themselves
|
|
|
|
|
* Why use synchronization when it is communication we need?
|
|
|
|
|
* Technology transfers badly to distibuted systems
|
|
|
|
|
* ... and this is before we start discussing how hard it is
|
|
|
|
|
|
|
|
|
|
**Why?**
|
|
|
|
|
|
|
|
|
|
* Part of the "real-time" design pattern
|
|
|
|
|
* "One thread per timing demand"
|
|
|
|
|
* We do have scheduling proofs and best practises
|
|
|
|
|
* Timing analysis is global anyway
|
|
|
|
|
* Scalability and deadlock analysis may not be the limiting constraint
|
|
|
|
|
* HW is shared memory architecture
|
|
|
|
|
* Infrastucture is avalible
|
|
|
|
|
* Communication systems requires infrastucture that we may not have
|
|
|
|
|
|
|
|
|
|
#### *All* resources are shared!
|
|
|
|
|
|
|
|
|
|
* Memory, certainly
|
|
|
|
|
* "Hidden" memory used by libraries (.. your own modules and the kernel)
|
|
|
|
|
* If the library takes care of this itself, it is called *"reentrant"*
|
|
|
|
|
* Sensors and actuators
|
|
|
|
|
* "CPU" - Computing capacity
|
|
|
|
|
* *This* is real-time programming; We solve it by *Scheduling*
|
|
|
|
|
* ... any other interface
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
#### Some standard problems/pit-falls
|
|
|
|
|
|
|
|
|
|
* **Race condition**: A bug that surfaces by unfortunate timing or order of events
|
|
|
|
|
* **Deadlock:** system in circular wait
|
|
|
|
|
* Special case of livelock
|
|
|
|
|
* Does not use CPU
|
|
|
|
|
* **Livelock:** system locked in a subset of states
|
|
|
|
|
* like deadlock, but we use CPU
|
|
|
|
|
* Busy-Waiting is a livelock
|
|
|
|
|
* **Starvation:** A thread does "by accident" not get the necessary resources
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
#### Features in syncronization
|
|
|
|
|
|
|
|
|
|
* Critical Section - Code that must not be interupted
|
|
|
|
|
* Mutual Exclusion - More piecesof code that must not interrupt each other
|
|
|
|
|
* Bounded buffer - Buffer with full/empty synchronization
|
|
|
|
|
* Read/Write Locks
|
|
|
|
|
* Readers can interleave eachother
|
|
|
|
|
* Writers have mutual exclusion
|
|
|
|
|
* Condition Syncronization - Blocking on event or status
|
|
|
|
|
* Guards etc.
|
|
|
|
|
* Resource allocation
|
|
|
|
|
* More than mutual exclution!
|
|
|
|
|
* Ref: The lock manager
|
|
|
|
|
* Rendezvouz/barriere - Synchronization point
|
|
|
|
|
* Ref: AA "end boundary"
|
|
|
|
|
* Communication
|
|
|
|
|
* Broadcast
|
|
|
|
|
* ...
|
|
|
|
|
|
|
|
|
|
|
