Pants Full of Unix

, you're gonna love this. Feel free to ping me tomorrow to guffaw.

Also, curiously, this builds on my last post, since it's an even-more complicated protocol (really, a set of protocols, but all in support of the same application) that needs to get through a firewall. Say what you want about how this might be better implemented with HTTP, webdav, gopher, or chargen: this shit's commercial software, kids, and it doesn't matter what you think… it needs to Just Work without modification to what the software wants to do beyond what the configuration lets you change.

While working on a time-senstive (and way overdue on that front) problem with a NetBackup SAN media server that has many issues (the acsssi daemon dumps core pseudo-randomly–not the issue in discussion here, though that one bears its own post at some point, and the path to configuring a NetBackup media server on the opposite side of a firewall from its master is… less than totally clear, including factually wrong instructions both in official documentation and in official tech notes), I sent email explaining my understanding of the firewall and related NetBackup configuration possibilities to one of the five (!!!) Veritas support reps I'm working with for this host.

I received this in response:

Date: 06/28/2005 02:14 PM
From: “Content Filtering”@[REDACTED]
To: “Undisclosed Recipient”@[REDACTED]
Subject: Profanity Blocking Of Email

Your recent electronic message was not delivered to its intended
recipient. The message appears to contain inappropriate material,
language or subject matter that is blocked. Please review your message
and if this is a business-related message and this message was blocked in
error, please remove any inappropriate material and resend the message.
Please refer to the [REDACTED] Electronic Media Policy at
http://www.security.[REDACTED].net if you have any questions regarding
inappropriate material. Please be advised that forwarding inappropriate
material, or any other violation of policy, could result in disciplinary
action up to and including employment termination.

For additional information, please contact your HR business partner.

Profanity: [REDACTED] Profanities
Message: Message Quarantined (Content Filtering)

Message details are as follows.

*****
From: [REDACTED]
Subject: Re: VERITAS Support: Case ID [REDACTED]
Date: 2005-06-28 13:14:15
*****

Here's my original message, with only identifying details removed:

Date: 06/28/2005 02:14 PM
From:
To: [REDACTED] < [REDACTED]@veritas.com>
Subject: Re: VERITAS Support: Case ID [REDACTED]

Just got your email about our existing firewall rules… I think I'll
indirectly cover what you wanted to double-check below, but if not, I'll
reply to it in a moment.

I want to make sure I understand the various cases, and ALL settings
necessary for them, in order to configure a media server on the other side
of a firewall from its master server. (It's not clear to me that either of
what was done before or what is kind of working now is actually the right
configuration for this host; and I want to make sure I understand the
possibilities so that I can pick the right one and put it in place.)

In these cases, my media server is a SAN media server, so configuration
specific to a client on the other side of a firewall that will always
write to a specific media server (not the master) isn't relevant for now.

Case 1: use vnetd for communication between the master server and the
media server.

I think I need some clarification about the CONNECT_OPTIONS setting.

My understanding is that on the master one should have:

CONNECT_OPTIONS media-svr 0 1 1

Then the media server should have:

CONNECT_OPTIONS master-svr 0 1 1

(I understand what the numbers do, but I'm sitting here looking at a
different media server, which I hope is misconfigured, where this line is
CONNECT_OPTIONS media-svr 0 1 1… I'm pretty sure that's Wrong, but I
want to be positive.)

These settings need to be made in both bp.conf and vm.conf. Additionally,
we need to set NO_CALLBACK 1 for this media server's name under db/clients.

The firewall rules I would need to request for this case are:

source host/port -> destination host/port
master/any -> media/13782 (bpcd)
master/any -> media/13701 (vmd)
media/any -> master/13724 (vnetd)
media/any -> master/13721 (bpdbm)
media/any -> master/13723 (bpjobd)
media/any -> master/13701 (vmd)
media/any -> acsls/30031
acsls/any -> media/30032

Case 2: use reserved ports for communication between the master server and
the media server.

If, instead, we wanted to use ports 512-1023, we would have those opened
in the firewall bi-directionally (as that is the default setting for both
CLIENT_RESERVED_PORT_WINDOW and SERVER_RESERVED_PORT_WINDOW), and we would
NOT want to have any CONNECT_OPTIONS settings for this configuration on
either host.

In this case, the firewall rules would be:

master/any -> media/13782 (bpcd)
master/any -> media/13701 (vmd)
media/any -> master/13721 (bpdbm)
media/any -> master/13723 (bpjobd)
media/any -> master/13701 (vmd)
master/512-1023 -> media/512-1023 (from CLIENT_RESERVED_PORT_WINDOW to SERVER_RESERVED_PORT_WINDOW)
media/512-1023 -> master/512-1023 (from CLIENT_RESERVED_PORT_WINDOW to SERVER_RESERVED_PORT_WINDOW)
media/any -> acsls/30031
acsls/any -> media/30032

Is this a workable configuration at all? That is, not opening any ports at
all on behalf of the CLIENT_ and SERVER_PORT_WINDOW settings? (Bear in
mind that I do need user-directed backups & restores to work, and I do
need restores to work initiated by EITHER end.) If not, then do Cases 2
and 3 (below) need to be merged for functionality?

Case 3: use non-reserved ports for communication between the master server
and the media server.

Set CLIENT_PORT_WINDOW to 4800-4899 on the media server ONLY.

Is it necessary to set ALLOW_NON_RESERVED_PORTS on the master? On the
media server?

Is it also necessary to set CONNECT_NON_RES_PORT to 1 in
db/client/media-svr on the master in this case?

master/any -> media/13782 (bpcd)
master/any -> media/13701 (vmd)
media/any -> master/13721 (bpdbm)
media/any -> master/13723 (bpjobd)
media/any -> master/13701 (vmd)
master/any -> media/1025-5000 (SERVER_PORT_WINDOW)
media/4800-4899 -> master/1025-5000 (SERVER_PORT_WINDOW)
media/any -> acsls/30031
acsls/any -> media/30032

One additional wrinkle:

Many people here commonly use the Windows NetBackup (rather than the Java
console X forwarded from the master) to perform NetBackup administration.
They do not do this on their workstation, they run it (via Terminal
Services) on one of our Windows media servers (which are on the same side
of the firewall as the master). Am I right in thinking that I need all the
same ports open between that Windows media server and the media server on
the other side of the firewall that I'm having opened for the master, or
do I only need a subset of those ports opened? Perhaps, only vmd?

–
[REDACTED]

Date: 06/27/2005 01:09 PM
From: [REDACTED] < [REDACTED]@veritas.com>
To:
Subject: VERITAS Support: Case ID [REDACTED]

Hey [REDACTED], FYI
Here is my boiler plate for firewall setup.
Please reply with the server's bp.conf, vm.conf, and the host files for
review.
Kind regards,
[REDACTED]

The 5.1 SAG starting on page 355 has details on clients using Vnetd-

VERITAS NetBackup ™ 5.1 System Administrators Guide for UNIX, Volume I
http://seer.support.veritas.com/docs/268090.htm

Additional details and tech note links-
For clients-
http://seer.support.veritas.com/docs/251630.htm
(note: a couple of caveats to this tech note 251630 for 5.0 and 5.1
clients.

1. In the NetBackup Administration Console, expand NetBackup Management >
Host
Properties > Master Servers > Double-click on master server > Client
Attributes.
2. In the client list, select the client you wish to change.
3. Under BPCD Connect-back, select VNETD Port.
4. Click OK. )

For media servers-
http://seer.support.veritas.com/docs/251631.htm

(note: a couple of caveats to this tech note 251631, first you must open
bpcd and Vnetd bi-directionally. Also you must configure the master server
to use Vnetd or no connect back in the media server host properties in the
firewall tab otherwise restores will fail. Also you will need to add the
same CONNECT_OPTIONS as seen in the bp.conf file to the vm.conf on both
servers.)

Here are a couple of ways to double check how the client is configured.
The output from running ./bpclient -client cdsntg02 -L is as follows:

Client Name: cdsntg02
Current Host:
Hostname: cdsntg02
IP Address: xxx.xxx.xxx.xxx
Connect on non-reserved port: no
No call-back connections: yes (means it is using Vnetd)
Dynamic Address: no
Free Browse: Deny
List Restore: Allow Both
Max Jobs This Client: 5

Here is the /netbackup/db/client/cdsntg02/host_info file
cat'd

CONNECT_NON_RES_PORT 0
DYNAMIC_ADDRESS 0
FREE_BROWSE 1
LIST_RESTORE 1
MAX_JOBS_THIS_CLIENT 5
NO_CALLBACK 1 (the `1' means it is using Vnetd. A `0' would be the default)

To change the client from the command line, you could use the bpclient
command as follows-
bpclient -client (client name) -update -no_callback 1

For server you should see a CONNECT_OPTION = (media server hostname) 0 1 1
In the servers bp.conf file. The second `1' means to use vnetd.

to troubleshoot the connection-
Run these on the client(s)bpclntcmd -ip (ip address of the server)
bpclntcmd -hn (hostname of the server)
bpclntcmd -pn (this should force the
server to do gethostbyaddress and verify the server sees the client
correctly but requires the bprd port 13720 to be opened on the firewall)

You can also try telneting from the master/media server to the client via
the bpcd port 13782. And then from the client to the server's bprd port
13720.
Use port 13724 if you are using Vnetd.
If this fails, then the firewall is still blocking the connection.

If all else fails, gather the following logs w/verbose and forward to me
please-
- From the client- bpcd and bpbkar
- From the media server- bpbrm and bptm

[REDACTED]

I tried sending the same information as a .txt attachment, and received the same filter message. I tried sending the same thing as a password-protected zip archive, and the mail server at the Veritas end quarantined the attachment, stating that it was unsafe as it was probably a virus (can't argue, really).

My best guesses for the filtering reason are repeated use of the word “master” (which is bullshit, since any inappropriate message with that characteristic would probably have plenty of other words that aren't commonly used to describe server/client computer relationships on which to filter) or the inclusion of “bi” (which is pretty questionable, and begging for a lawsuit over discrimination from the LGBT community with ACLU backing, a lawsuit I'd whole-heartedly support).

For those of you playing along at home with NetBackup…

In case 1:

• I'm right that the CONNECT_OPTIONS host argument should be reciprocal (list the master on the media server, list the media server on the master),
• NO_CALLBACK in the client properties for the media server's name (in db/clients/media-svr on the master) needs to be set to 1 in order for user-directed (that is, application API, so DB2 here, but also Oracle, so forth) backups & restores to work and also so that it's possible to fall back to running backups for this system as a regular client in the event that its SAN or SCSI access to tape is interrupted,
• the firewall rules permitting traffic with a destination port of bpcd, vmd, and vnetd are necessary in both directions for communication between a master and a media server,
• and the firewall rules permitting traffic with a destination port of bpdbm and bpjobd are not necessary under the vnetd configuration as of NetBackup 5, as communication for those daemons was merged into the vnetd protocol with that major version (but I wouldn't trust it before 5.1).

In case 2:

• Again, bpcd and vmd need to be open bi-directionally (vnetd is not used here),
• and this is a workable configuration, as NetBackup will not try to use non-reserved ports unless told to; no merging of configuration from case 3 is necessary.

In case 3:

• It is not the intended use to set CLIENT_PORT_WINDOW only on one end of the connection, but this configuration will do what I suggest it will do, as it controls which source ports will be used for connections (SERVER_PORT_WINDOW controls which destination ports will be used), meaning the setting is only necessary if your firewall filters based on source port (which I happen to think all should, but never mind that); 100 ports in the window may not be sufficient for a given high-traffic server, even one that's writing directly to tape rather than sending its backup data across the network, as each backup or restore needs somewhere between 3 and 5 sockets depending on what it's doing,
• it is necessary to set ALLOW_NON_RESERVED_PORTS on the master if the media server (or client) is intended to connect to ports in the range SERVER_PORT_WINDOW (1025-5000 here and by default),
• it is necessary to set CONNECT_NON_RES_PORT to 1 in db/client/media-svr on the master in order for connections back from the master to the media server/client in response to user-directed backups & restores to get through the firewall as described,
• and again, bpcd and vmd should be open both ways, again vnetd is not used in this case.

For the additional wrinkle, it's necessary to open all the same ports for communication between the Windows media server/admin host and the media server on the other side of the firewall with the exception of bpdbm and bpjobd, which daemons are only listing on the master server in a given environment.