It’s tempting to use short mnemonic text tags in the summary field of your bug tracking system - you know, a tag to indicate that this bug only appears in a short-lived branch, or applies only to a specific patch to a particular customer - the uses are endless.
Don’t do it. Just stop. The reason is simple. Each time you ask the other members of your team to apply some arbitrary tag to certain issues in the bug tracking system, you devalue your bug tracking system.
Why? Because you’re probably misusing your bug tracker’s search feature. Chances are the members of your team are already assiduously storing the information that you need; branch, build number, version, product, component - it’s probably all there. Spend a bit more time figuring out how to construct the search you need - you probably don’t need that extra free-text field!
If you realise you don’t need that field after all, you’ve just saved your team members and yourself a bunch of hassle. Think about it; You can trust folks to accurately choose an option from a drop-down list, especially if it’s a required field, but if you ask folks to add a tag like “AcmeCorp” to their bug reports, how many of them will remember to do it? How many of them will spell the tag just the way you expect?
When you attempt to search on the tag, you won’t be able to trust the results you get back, because Ted spelled ‘Acme’ as ‘Amce’ and Diana just forgot to add the tag. What’s the point of that? Just learn to use your bug tracker’s search form like a pro.
April 6, 2010
“…like watching paint dry”
Colleagues who volunteer comments like “testing is more boring than watching paint dry!” make me laugh. To me that displays an astonishing lack of curiosity. Yes, watching paint dry is boring, but only if you’re entirely passive.
Testing is as interesting as you want to make it. Let’s imagine testing really is like watching paint dry. What sort of questions can you ask about paint that’s drying? From those questions, what kinds of tests could you devise to help you find out new things about the process?
- How does the paint dry? From the edges in, or from the surface down?
- If it’s from the surface down, does a skin form?
- Is that skin permeable to air? If not, how can the lower layer of paint dry?
- Does the crust inhibit drying? Could paint be formulated to dry more quickly?
- What’s our definition of ‘dry’? Some maximum percentage moisture content?
- How can I observe the drying process without interfering with that process?
- Do oil-based paints dry differently to water-based or emulsion paints?
- At what level of detail can we be said to be watching paint dry? Are we attempting to observe individual H20 molecules being liberated from the paint?
If you’ve got an inquisitive mind and a bit of a scientific bent, you’ll see that questions like these provide fodder for a variety of tests which will undoubtedly lead you to interesting discoveries.
I think you’ll find that paint dries surprisingly fast when your brain is engaged! Now, what about watching grass grow..?
February 17, 2010
Working with filenames containing spaces in Bash
The problem: spaces in filenames
Sometimes you’ll need to manipulate a list of filenames which contain spaces using a ‘for’ loop. Let’s say you have the following two files:
Command:
$ ls -1
Output:
Aisling, Sanda and Justin.jpg
Monika, Sanda, Justin, Michelle and Ben.jpg
They list perfectly well - one file on each line. (I used the ‘-1’ [minus one] option to ‘ls’ to make a single-column list)
However something strange happens when you want to reference them individually for a ‘for’ loop. Here I use a ‘for’ loop to see how the filenames are being passed to the ‘echo’ command:
Command:
$ for line in $(ls -1) ; do echo ${line} ; done
Output:
Aisling,
Sanda
and
Justin.jpg
Monika,
Sanda,
Justin,
Michelle
and
Ben.jpg
Aaagh! The filenames are being chopped up at the spaces and at the newlines!
The solution: a custom value for $IFS
I need to be able to tell bash that I only want it to chop up lists at the newline character. For this, I modify the value of the built-in variable, IFS. The name stands for ‘Internal Field Separator’.
IFS normally has the value ” ” - that’s to say, space, tab and newline. Let’s reset it so that it only contains the newline character:
$ IFS=$‘’
See how I specify the newline character? That’s important. It’s different from how you’d normally set a shell variable. I don’t want the literal string ‘’, I want what that string represents - a newline.
Now I run the same ‘for’ loop again:
Command:
$ for line in $(ls -1) ; do echo ${line} ; done
Output:
Aisling, Sanda and Justin.jpg
Monika, Sanda, Justin, Michelle and Ben.jpgNow I can manipulate the filenames as I originally intended.
An example: get rid of those spaces!
Here’s a shell script that uses the ‘tr’ command to rename a list of files, replacing tab and space characters in filenames with underscores:
#!/bin/bash
# Back up the current value of IFS
IFS=$IFS
IFS=$‘’
for line in $(ls -1) ; do
newname=$(echo ${line} | tr ’ ’ ’’)
escapedname=$(echo ${line} | tr ’ ’ “ ”)
mv ${escapedname} ${newname}
done
# Restore IFS
IFS=$_IFS
Here’s what my files look like after I’ve run that script:
Aisling,_Sanda_and_Justin.jpg
Monika,_Sanda,_Justin,_Michelle_and_Ben.jpg
August 7, 2009
Using Oracle Import and Export
Summary
This is a description on how to use Oracle’s exp and imp utilities to move tables between databases. It transfers the data over the network, which avoids the requirement to store an intermediate file and also neatly side-steps the Linux 32Gb file limit for big databases. It’s a good idea to run the Oracle commands in a screen session.
Create a database account on the destination host which matches the name of the account on the source host
drop user owner cascade
create user owner identified by password
grant connect, resource, create view to owner
Allow no-password ssh logins from the source host to the destination host
As the oracle user on the source database host:
- Run
ssh-keygen on the source host. Don’t specify a passphrase.
$ ssh-keygen -t rsa
- Copy the public key you just generated to the destination machine:
$ ssh-copy-id oracle@dest-host
Set up a named pipe on the destination host
Logged in a oracle on the destination host:
$ mknod /tmp/inpipe.dmp p
Set oracle imp to read data from that pipe
Destination host:
$ imp system/password@orcl fromuser=ppowner touser=ppowner file=/tmp/inpipe.dmp &
Set up a named pipe on the source host
Source host:
$ mknod /tmp/outpipe.dmp p
Allow the pipe to be tunnelled over ssh
Soure host:
ssh -C oracle@dest-host 'cat >/tmp/inpipe.dmp' < /tmp/outpipe.dmp &
Set oracle exp to write to the named pipe
Source host:
$ exp system/password@orcl owner=ppowner file=/tmp/outpipe.dmp
You should expect to see output from exp and imp on both hosts. The export process may finish before the import; that’s OK, just leave the import to finish in its own sweet time.
Exporting and Importing individual database users
Run the following command to export the data for the user owner, to the owner.dmp file:
exp userid=owner/owner grants=y indexes=y file=owner.dmp
For each user, the username, password and the export dump file should be changed accordingly.
Do the following to import all the data for a user:
- Make sure the user being imported exists in the database, with ownership to the same table spaces as before (when the user data was exported).
- Run the following command to import the data. It’s virtually the same command which exported the data. The
exp command is replaced with the imp command. The example below imports all the data belonging to the owner user into the new database.
$ imp userid=owner/owner grants=y indexes=y statistics=recalculate file=owner.dmp
A caveat: imp and exp can cause stale statistics to be imported.
During functional and performance testing, I regularly use Oracle imp and exp to backup and restore database snapshots. If you use these tools, you need to be aware that the restored snapshot may exhibit different SQL performance characteristics than the original database.
The effect on performance has to do with re-importing questionable statistics: From http://www.lc.leidenuniv.nl/awcourse/oracle/server.920/a96525/expus.htm
EXP-00091 Exporting questionable statistics
Cause: Export was able to export statistics, but the statistics may not be useable. The statistics are questionable because one or more of the following happened during export: a row error occurred, client character set or NCHARSET does not match with the server, a query clause was specified on export, only certain partitions or subpartitions were exported, or a fatal error occurred while processing a table.
Action: To export non-questionable statistics, change the client character set or NCHARSET to match the server, export with no query clause, or export complete tables. If desired, import parameters can be supplied so that only non-questionable statistics will be imported, and all questionable statistics will be recalculated.
The following parameter to imp forces all statistics to be recalculated on import: statistics=recalculate
Getting around IMP-00013: only a DBA can import a file exported by another DBA
- give your user import/export full_database rights grant imp_full_database,exp_full_database to ppowner36;
- User import FROMUSER TOUSER imp owner/owner file=owner.dmp FROMUSER=OWNERA TOUSER=OWNERB; Where OWNERA is the user who originally owned the data. OWNERB is the new user who you want to import the data to.
November 13, 2008
Discover Linux System Resources
You’ve just inherited a Linux machine and you need to know how many CPUs it has, what speed they go at, how many memory slots there are and what’s in them, not to mention finding out what video card is installed so you can install a closed-source driver from the card manufacturer, but you don’t want to shut down everything and open the case. What can you do? Linux provides a few very useful utilities to discover exactly what’s inside the box.
lspci
This command lists all of the devices attached to the system’s PCI bus. Use it to discover what video card is installed, for example. It has several verbosity levels - see the man page. lspci is provided by the package pciutils.i386.
$ /sbin/lspci
00:00.0 Host bridge: Broadcom CMIC-LE Host Bridge (GC-LE chipset) (rev 33)
00:00.1 Host bridge: Broadcom CMIC-LE Host Bridge (GC-LE chipset)
00:00.2 Host bridge: Broadcom CMIC-LE Host Bridge (GC-LE chipset)
00:03.0 VGA compatible controller: ATI Technologies Inc Rage XL (rev 27)
00:04.0 System peripheral: Compaq Computer Corporation Integrated Lights Out Controller (rev 01)
00:04.2 System peripheral: Compaq Computer Corporation Integrated Lights Out Processor (rev 01)
…
dmidecode
Use dmidecode to find out exactly what what memory is installed in your computer, which banks it’s in, how many banks you have and the speed and type of that memory. dmidecode is pretty verbose by default, so pipe its output through less. You need to be root (or have sudo privileges) to run dmidecode. You can also use dmidecode to enumerate all of the CPUs in your system, even if one or more are disabled. dmidecode is provided by the package kernel-utils.i386.
$ sudo /usr/sbin/dmidecode | less
…
Handle 0x1100
DMI type 17, 23 bytes.
Memory Device
Array Handle: 0x1000
Error Information Handle: Not Provided
Total Width: 72 bits
Data Width: 64 bits
Size: 1024 MB
Form Factor: DIMM
Set: 1
Locator: DIMM 01
Bank Locator: Not Specified
Type: DDR
Type Detail: Synchronous
Speed: 266 MHz (3.8 ns)
…
/proc/cpuinfo
To find out exactly what CPUs are currently enabled and what speed they’re running at, take a look at the contents of the proc filesystem file /proc/cpuinfo. Note the rated speed (model name) and the actual speed (cpu MHz) at which it’s running. /proc/cpuinfo is installed as part of the kernel. There are other useful virtual files under proc which can be explored.
$ cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 15
model : 2
model name : Intel(R) Xeon(TM) CPU 2.80GHz
stepping : 9
cpu MHz : 2792.114
cache size : 512 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe cid xtpr
bogomips : 5585.80
lshal
/usr/bin/lshal is a very useful command which provides information about every device visible to the Hardware Abstraction Layer, which on most Linuxes is every bit of hardware in your system. Use this to find exact details of Ethernet cards, hard disks, DVD-ROM drives… The utility is provided as part of the package hal.i386. Here’s a snippet which shows my exact motherboard make and model:
$ /usr/bin/lshal | head -52
Dumping 123 device(s) from the Global Device List:
————————————————-
udi = ‘/org/freedesktop/Hal/devices/computer’
info.addons = {‘hald-addon-cpufreq’, ‘hald-addon-acpi’} (string list)
info.bus = ‘unknown’ (string)
info.callouts.add = {‘hal-acl-tool –remove-all’, ‘hal-storage-cleanup-all-mountpoints’} (string list)
info.callouts.session_active = {‘hal-acl-tool –reconfigure’} (string list)
info.callouts.session_add = {‘hal-acl-tool –reconfigure’} (string list)
info.callouts.session_inactive = {‘hal-acl-tool –reconfigure’} (string list)
info.callouts.session_remove = {‘hal-acl-tool –reconfigure’} (string list)
info.interfaces = {‘org.freedesktop.Hal.Device.SystemPowerManagement’} (string list)
info.product = ‘Computer’ (string)
info.subsystem = ‘unknown’ (string)
info.udi = ‘/org/freedesktop/Hal/devices/computer’ (string)
org.freedesktop.Hal.Device.SystemPowerManagement.method_argnames = {‘num_seconds_to_sleep’, ‘num_seconds_to_sleep’, ’‘,’‘,’‘, ’enable_power_save’} (string list)
org.freedesktop.Hal.Device.SystemPowerManagement.method_execpaths = {‘hal-system-power-suspend’, ‘hal-system-power-suspend-hybrid’, ‘hal-system-power-hibernate’, ‘hal-system-power-shutdown’, ‘hal-system-power-reboot’, ‘hal-system-power-set-power-save’} (string list)
org.freedesktop.Hal.Device.SystemPowerManagement.method_names = {‘Suspend’, ‘SuspendHybrid’, ‘Hibernate’, ‘Shutdown’, ‘Reboot’, ‘SetPowerSave’} (string list)
org.freedesktop.Hal.Device.SystemPowerManagement.method_signatures = {‘i’, ‘i’, ’‘,’‘,’‘, ’b’} (string list)
power_management.acpi.linux.version = ‘20070126’ (string)
power_management.can_hibernate = true (bool)
power_management.can_suspend = true (bool)
power_management.can_suspend_hybrid = false (bool)
power_management.can_suspend_to_disk = true (bool)
power_management.can_suspend_to_ram = true (bool)
power_management.is_powersave_set = false (bool)
power_management.type = ‘acpi’ (string)
smbios.bios.release_date = ‘03/22/2008’ (string)
smbios.bios.vendor = ‘Dell Inc.’ (string)
smbios.bios.version = ‘A02’ (string)
smbios.chassis.manufacturer = ‘Dell Inc.’ (string)
smbios.chassis.type = ‘Tower’ (string)
smbios.system.manufacturer = ‘Dell Inc.’ (string)
smbios.system.product = ‘Precision WorkStation T5400’ (string)
smbios.system.serial = ‘BBXFP3J’ (string)
smbios.system.uuid = ‘44454C4C-4200-1058-8046-C2C04F50334A’ (string)
system.chassis.manufacturer = ‘Dell Inc.’ (string)
system.chassis.type = ‘Tower’ (string)
system.firmware.release_date = ‘03/22/2008’ (string)
system.firmware.vendor = ‘Dell Inc.’ (string)
system.firmware.version = ‘A02’ (string)
system.formfactor = ‘desktop’ (string)
system.hardware.primary_video.product = 1039 (0x40f) (int)
system.hardware.primary_video.vendor = 4318 (0x10de) (int)
system.hardware.product = ‘Precision WorkStation T5400’ (string)
system.hardware.serial = ‘BBXFP3J’ (string)
system.hardware.uuid = ‘44454C4C-4200-1058-8046-C2C04F50334A’ (string)
system.hardware.vendor = ‘Dell Inc.’ (string)
system.kernel.machine = ‘x86_64’ (string)
system.kernel.name = ‘Linux’ (string)
system.kernel.version = ‘2.6.25.14-69.fc8’ (string)
…
hdparm
Use hdparm to discover a wealth of information about installed disks. You need root access or sudo privileges to run the command, and it’s pretty verbose. hdparm worn’t work for devices connected to a SMART array (a type of disk controller) (Use smartctl if hdparm returns the error HDIO_GET_MULTCOUNT failed: Inappropriate ioctl for device)
$ sudo /sbin/hdparm -I /dev/sda
/dev/sda:
ATA device, with non-removable media
Model Number: Hitachi HDP725050GLA360
Serial Number: GEA550RE27DYHL
Firmware Revision: GM4OA5BA
Transport: Serial, ATA8-AST, SATA 1.0a, SATA II Extensions, SATA Rev 2.5; Revision: ATA8-AST T13 Project D1697 Revision 0b
Standards:
Supported: 8 7 6 5
Likely used: 8
Configuration:
Logical max current
cylinders 16383 16383
heads 16 16
sectors/track 63 63
–
CHS current addressable sectors: 16514064
LBA user addressable sectors: 268435455
LBA48 user addressable sectors: 976773168
device size with M = 10241024: 476940 MBytes
device size with M = 10001000: 500107 MBytes (500 GB)
…
dumpe2fs
This invaluable command is one of the only ways to discover your disk’s block size - essential for being able to figure out what vmstat’s “blocks per second” counter is trying to tell you.
$ sudo dumpe2fs /dev/sda1 | grep -i “block size”
dumpe2fs 1.39 (29-May-2006)
Block size: 1024
The answer tells you how many bytes in a block.
smarctl
Essential for finding out about connected SCSI devices. Right now I’m not using any boxes that use a SMART array controller. Take a look at the smartmontools article on the Gentoo wiki.
October 2, 2008
This blog is about Linux, testing and tech. I hope you find the solution to your problem here.
I’ve worked as a QA Engineer in a variety of software companies in Ireland since 1996. I’ve dabbled with Linux since about 1998. In recent years, I’ve used it as my desktop at work (since about 2000) and since 2007 all of my boxes at home run some variant of Linux (mostly Ubuntu) — no more Windows anywhere.
If you want to get in touch for any reason, drop me an email at this domain or search for me on LinkedIn or Twitter.
regards
Ben
September 12, 2008