Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.XX..X..XX....X...X.XX.XX.......X..X......XXXX.....XX.X.....] duration: 5.966286631s trace: https://jaeger.dev.storj.io/trace/4472a4ab736f5ac FAIL (2025-12-28 09:01:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 08:57:38.644991590 (545h28m38.516074635s) [upload=478331]: Starting upload Dec 28 08:57:38.645074659 (545h28m38.516157704s) [upload=478331]: Done starting upload Dec 28 08:57:38.645239703 (545h28m38.516322738s) [upload=478331]: Uploading segments... Dec 28 08:57:38.645381684 (545h28m38.516464729s) [upload=478331]: Got next segment. Inline: false Dec 28 08:57:38.645426229 (545h28m38.516509264s) [upload=478331]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 08:57:38.645465563 (545h28m38.516548608s) [upload=478331]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 08:57:38.894465663 (545h28m38.765548718s) [seg_pos={0 0}, upload=478331]: Begin upload segment... Dec 28 08:57:38.894532079 (545h28m38.765615114s) [seg_pos={0 0}, upload=478331]: Done begin upload segment. Dec 28 08:57:38.894574179 (545h28m38.765657214s) [noise=true , address=145.79.227.50:4373 , piece_id=QT3FSYZYIGJIPOHPO2L3VHG2AXPPNV4KVRJ5OPIDJ7T43RKMYKKQ, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.894676474 (545h28m38.765759519s) [noise=true , address=145.79.227.45:4229 , piece_id=FZ7D676G2K3DWTMANEAICJU7ECJZWRDLTNCFFKTDFEFGT7ADDX7A, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.895532635 (545h28m38.76661567s) [noise=true , address=77.90.31.19:4844 , piece_id=F3LZ7EJOCTGNF4GJBX26AGC4QB4WQ6QHSSXJ6AOQWBXRMZSL34ZA, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.895644868 (545h28m38.766727913s) [noise=true , address=45.91.116.9:4261 , piece_id=MMTDQLCP43W5LDA6BGDPVRBQ4E2VQ7GBEPDYOUPXFCDPX3ORRPYA, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.896029460 (545h28m38.767112495s) [noise=true , address=45.91.116.7:4010 , piece_id=PICVBYU3LMFEJ2UTHVQVHMQQB3EYZ25XXAGM6DELW5DLDJ7CHJOQ, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.896322279 (545h28m38.767405314s) [noise=true , address=145.79.227.3:4151 , piece_id=Z4AZOB7YRU54ALQIXNHV7K5YXBRQ2VOHPYWAYI2PVK4FZG2VCHAQ, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.896682224 (545h28m38.767765269s) [noise=true , address=145.79.227.51:4409 , piece_id=TC6HJ3S7VM6OA6BEXZGM6V43WT7VY7ZP7XTM2ZUIKWAVPCXCPV7Q, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.896720056 (545h28m38.767803101s) [noise=true , address=145.79.227.29:4942 , piece_id=ZMNDBC6VLZEBJ2WJ66EX3ML7VHFW7IQLPQTU5EMWIKVDEHIYKVTA, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.897026940 (545h28m38.768109985s) [noise=true , address=145.79.227.27:4691 , piece_id=RYCQQN4SLZGLJ4NDSLPXZGDSAXCEDC3PYMQ7KW7NJKFW22COSCPA, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.897134114 (545h28m38.768217149s) [noise=true , address=145.79.227.11:4731 , piece_id=KIWK6CLD5OHQ7XEOHPURMMPIP6WCHUNDVQR7KYTO73D5JYM5K37Q, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.897431581 (545h28m38.768514626s) [noise=true , address=145.79.227.39:4013 , piece_id=MCN5VHIKQTFS2WY6NENYVETCIQ5COTMHDMW4BO3FS56ISCIR2IXQ, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.897624919 (545h28m38.768707954s) [noise=true , address=45.91.116.11:4119 , piece_id=5P3HKX2KJRS4GYU2GYWJ5FUAXEWVOB5EIWM3RND6MY3NOMPIBFXA, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.897872700 (545h28m38.768955745s) [noise=true , address=145.79.227.41:4087 , piece_id=GB226GWOX7TC23BAQTUJN6QU2VO7QIONZ5VSJI4NY5ESARCH7WYA, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.898120582 (545h28m38.769203617s) [noise=true , address=45.91.116.5:4297 , piece_id=NQYN7AQNQNF7Z4TJJYUCXKPVVC5I3JXXPPKA3H4V2FJDBL2GBG2A, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.898505405 (545h28m38.76958844s) [noise=true , address=145.79.227.9:4365 , piece_id=NAU2HWKLTB7NHLNBGBMHFLTL6S2BH4OPLO6FWWELH2DSPZ6MEZNQ, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.898901520 (545h28m38.769984555s) [noise=true , address=145.79.227.17:4971 , piece_id=TAUKFPKSMBYPJM73TXEJJHMYM7ADJW57ZQF4V2ALOU3TJOBVHH4A, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:40.609896701 (545h28m40.480979736s) [upload=478331]: Next returned nil segment Dec 28 08:57:40.609912210 (545h28m40.480995245s) [upload=478331]: Waiting for error group managing segments... Dec 28 08:57:40.894791250 (545h28m40.765874295s) [noise=true , address=145.79.227.27:4691 , piece_id=RYCQQN4SLZGLJ4NDSLPXZGDSAXCEDC3PYMQ7KW7NJKFW22COSCPA, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.901466881 (545h28m40.772549916s) [noise=true , address=145.79.227.50:4373 , piece_id=QT3FSYZYIGJIPOHPO2L3VHG2AXPPNV4KVRJ5OPIDJ7T43RKMYKKQ, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.902367305 (545h28m40.77345035s) [noise=true , address=145.79.227.11:4731 , piece_id=KIWK6CLD5OHQ7XEOHPURMMPIP6WCHUNDVQR7KYTO73D5JYM5K37Q, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.903421053 (545h28m40.774504088s) [noise=true , address=145.79.227.39:4013 , piece_id=MCN5VHIKQTFS2WY6NENYVETCIQ5COTMHDMW4BO3FS56ISCIR2IXQ, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.905881196 (545h28m40.776964232s) [noise=true , address=145.79.227.41:4087 , piece_id=GB226GWOX7TC23BAQTUJN6QU2VO7QIONZ5VSJI4NY5ESARCH7WYA, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.908557673 (545h28m40.779640708s) [noise=true , address=145.79.227.9:4365 , piece_id=NAU2HWKLTB7NHLNBGBMHFLTL6S2BH4OPLO6FWWELH2DSPZ6MEZNQ, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.909631207 (545h28m40.780714253s) [noise=true , address=145.79.227.29:4942 , piece_id=ZMNDBC6VLZEBJ2WJ66EX3ML7VHFW7IQLPQTU5EMWIKVDEHIYKVTA, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.910935481 (545h28m40.782018516s) [noise=true , address=145.79.227.45:4229 , piece_id=FZ7D676G2K3DWTMANEAICJU7ECJZWRDLTNCFFKTDFEFGT7ADDX7A, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.913792060 (545h28m40.784875095s) [noise=true , address=145.79.227.17:4971 , piece_id=TAUKFPKSMBYPJM73TXEJJHMYM7ADJW57ZQF4V2ALOU3TJOBVHH4A, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.917421171 (545h28m40.788504216s) [noise=true , address=145.79.227.51:4409 , piece_id=TC6HJ3S7VM6OA6BEXZGM6V43WT7VY7ZP7XTM2ZUIKWAVPCXCPV7Q, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.934456098 (545h28m40.805539133s) [noise=true , address=145.79.227.3:4151 , piece_id=Z4AZOB7YRU54ALQIXNHV7K5YXBRQ2VOHPYWAYI2PVK4FZG2VCHAQ, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:41.081958449 (545h28m40.953041494s) [noise=true , address=77.90.31.19:4844 , piece_id=F3LZ7EJOCTGNF4GJBX26AGC4QB4WQ6QHSSXJ6AOQWBXRMZSL34ZA, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:41.131461521 (545h28m41.002544576s) [noise=true , address=45.91.116.9:4261 , piece_id=MMTDQLCP43W5LDA6BGDPVRBQ4E2VQ7GBEPDYOUPXFCDPX3ORRPYA, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:41.131488533 (545h28m41.002571588s) [seg_pos={0 0}, upload=478331]: Segment reached optimal threshold of 13 pieces. Dec 28 08:57:41.131771992 (545h28m41.002855038s) [noise=true , address=45.91.116.11:4119 , piece_id=5P3HKX2KJRS4GYU2GYWJ5FUAXEWVOB5EIWM3RND6MY3NOMPIBFXA, seg_pos={0 0}, upload=478331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 28 08:57:41.131782632 (545h28m41.002865677s) [noise=true , address=45.91.116.11:4119 , piece_id=5P3HKX2KJRS4GYU2GYWJ5FUAXEWVOB5EIWM3RND6MY3NOMPIBFXA, seg_pos={0 0}, upload=478331]: Long tail cancellation due to optimal threshold Dec 28 08:57:41.131968947 (545h28m41.003052012s) [noise=true , address=45.91.116.7:4010 , piece_id=PICVBYU3LMFEJ2UTHVQVHMQQB3EYZ25XXAGM6DELW5DLDJ7CHJOQ, seg_pos={0 0}, upload=478331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:44598->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:44598->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44598->45.91.116.7:4010: use of closed network connection Dec 28 08:57:41.132006558 (545h28m41.003089623s) [noise=true , address=45.91.116.7:4010 , piece_id=PICVBYU3LMFEJ2UTHVQVHMQQB3EYZ25XXAGM6DELW5DLDJ7CHJOQ, seg_pos={0 0}, upload=478331]: Long tail cancellation due to optimal threshold Dec 28 08:57:41.132100998 (545h28m41.003184034s) [noise=true , address=45.91.116.5:4297 , piece_id=NQYN7AQNQNF7Z4TJJYUCXKPVVC5I3JXXPPKA3H4V2FJDBL2GBG2A, seg_pos={0 0}, upload=478331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 28 08:57:41.132113924 (545h28m41.003196959s) [noise=true , address=45.91.116.5:4297 , piece_id=NQYN7AQNQNF7Z4TJJYUCXKPVVC5I3JXXPPKA3H4V2FJDBL2GBG2A, seg_pos={0 0}, upload=478331]: Long tail cancellation due to optimal threshold Dec 28 08:57:41.132140273 (545h28m41.003223318s) [seg_pos={0 0}, upload=478331]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 08:57:41.132230986 (545h28m41.003314031s) [upload=478331]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 08:57:41.132238360 (545h28m41.003321405s) [upload=478331]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 08:57:41.428285145 (545h28m41.29936818s) [upload=478331]: Done uploading segments... Dec 28 08:57:41.428349467 (545h28m41.299432513s) [upload=478331]: Upload finished. err: <nil> Duration: 5.966312521s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [.XX.......XX.X....XXXX.XX....X....X.XX..XX..X........X.X...X] duration: 5.001465553s trace: https://jaeger.dev.storj.io/trace/6edfaf8a0832c07a FAIL (2025-12-28 09:00:53 UTC) orbiter warning: context deadline exceeded Duration: 5.001501751s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X.XX..X..XX....X...X.XX.XX.......X..X......XXXX.....XX.X....] duration: 5.686258668s trace: https://jaeger.dev.storj.io/trace/4472a4ab736f5ac FAIL (2025-12-28 09:00:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 08:57:38.644991590 (545h28m38.516074635s) [upload=478331]: Starting upload Dec 28 08:57:38.645074659 (545h28m38.516157704s) [upload=478331]: Done starting upload Dec 28 08:57:38.645239703 (545h28m38.516322738s) [upload=478331]: Uploading segments... Dec 28 08:57:38.645381684 (545h28m38.516464729s) [upload=478331]: Got next segment. Inline: false Dec 28 08:57:38.645426229 (545h28m38.516509264s) [upload=478331]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 08:57:38.645465563 (545h28m38.516548608s) [upload=478331]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 08:57:38.894465663 (545h28m38.765548718s) [seg_pos={0 0}, upload=478331]: Begin upload segment... Dec 28 08:57:38.894532079 (545h28m38.765615114s) [seg_pos={0 0}, upload=478331]: Done begin upload segment. Dec 28 08:57:38.894574179 (545h28m38.765657214s) [noise=true , address=145.79.227.50:4373 , piece_id=QT3FSYZYIGJIPOHPO2L3VHG2AXPPNV4KVRJ5OPIDJ7T43RKMYKKQ, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.894676474 (545h28m38.765759519s) [noise=true , address=145.79.227.45:4229 , piece_id=FZ7D676G2K3DWTMANEAICJU7ECJZWRDLTNCFFKTDFEFGT7ADDX7A, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.895532635 (545h28m38.76661567s) [noise=true , address=77.90.31.19:4844 , piece_id=F3LZ7EJOCTGNF4GJBX26AGC4QB4WQ6QHSSXJ6AOQWBXRMZSL34ZA, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.895644868 (545h28m38.766727913s) [noise=true , address=45.91.116.9:4261 , piece_id=MMTDQLCP43W5LDA6BGDPVRBQ4E2VQ7GBEPDYOUPXFCDPX3ORRPYA, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.896029460 (545h28m38.767112495s) [noise=true , address=45.91.116.7:4010 , piece_id=PICVBYU3LMFEJ2UTHVQVHMQQB3EYZ25XXAGM6DELW5DLDJ7CHJOQ, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.896322279 (545h28m38.767405314s) [noise=true , address=145.79.227.3:4151 , piece_id=Z4AZOB7YRU54ALQIXNHV7K5YXBRQ2VOHPYWAYI2PVK4FZG2VCHAQ, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.896682224 (545h28m38.767765269s) [noise=true , address=145.79.227.51:4409 , piece_id=TC6HJ3S7VM6OA6BEXZGM6V43WT7VY7ZP7XTM2ZUIKWAVPCXCPV7Q, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.896720056 (545h28m38.767803101s) [noise=true , address=145.79.227.29:4942 , piece_id=ZMNDBC6VLZEBJ2WJ66EX3ML7VHFW7IQLPQTU5EMWIKVDEHIYKVTA, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.897026940 (545h28m38.768109985s) [noise=true , address=145.79.227.27:4691 , piece_id=RYCQQN4SLZGLJ4NDSLPXZGDSAXCEDC3PYMQ7KW7NJKFW22COSCPA, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.897134114 (545h28m38.768217149s) [noise=true , address=145.79.227.11:4731 , piece_id=KIWK6CLD5OHQ7XEOHPURMMPIP6WCHUNDVQR7KYTO73D5JYM5K37Q, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.897431581 (545h28m38.768514626s) [noise=true , address=145.79.227.39:4013 , piece_id=MCN5VHIKQTFS2WY6NENYVETCIQ5COTMHDMW4BO3FS56ISCIR2IXQ, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.897624919 (545h28m38.768707954s) [noise=true , address=45.91.116.11:4119 , piece_id=5P3HKX2KJRS4GYU2GYWJ5FUAXEWVOB5EIWM3RND6MY3NOMPIBFXA, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.897872700 (545h28m38.768955745s) [noise=true , address=145.79.227.41:4087 , piece_id=GB226GWOX7TC23BAQTUJN6QU2VO7QIONZ5VSJI4NY5ESARCH7WYA, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.898120582 (545h28m38.769203617s) [noise=true , address=45.91.116.5:4297 , piece_id=NQYN7AQNQNF7Z4TJJYUCXKPVVC5I3JXXPPKA3H4V2FJDBL2GBG2A, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.898505405 (545h28m38.76958844s) [noise=true , address=145.79.227.9:4365 , piece_id=NAU2HWKLTB7NHLNBGBMHFLTL6S2BH4OPLO6FWWELH2DSPZ6MEZNQ, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.898901520 (545h28m38.769984555s) [noise=true , address=145.79.227.17:4971 , piece_id=TAUKFPKSMBYPJM73TXEJJHMYM7ADJW57ZQF4V2ALOU3TJOBVHH4A, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:40.609896701 (545h28m40.480979736s) [upload=478331]: Next returned nil segment Dec 28 08:57:40.609912210 (545h28m40.480995245s) [upload=478331]: Waiting for error group managing segments... Dec 28 08:57:40.894791250 (545h28m40.765874295s) [noise=true , address=145.79.227.27:4691 , piece_id=RYCQQN4SLZGLJ4NDSLPXZGDSAXCEDC3PYMQ7KW7NJKFW22COSCPA, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.901466881 (545h28m40.772549916s) [noise=true , address=145.79.227.50:4373 , piece_id=QT3FSYZYIGJIPOHPO2L3VHG2AXPPNV4KVRJ5OPIDJ7T43RKMYKKQ, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.902367305 (545h28m40.77345035s) [noise=true , address=145.79.227.11:4731 , piece_id=KIWK6CLD5OHQ7XEOHPURMMPIP6WCHUNDVQR7KYTO73D5JYM5K37Q, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.903421053 (545h28m40.774504088s) [noise=true , address=145.79.227.39:4013 , piece_id=MCN5VHIKQTFS2WY6NENYVETCIQ5COTMHDMW4BO3FS56ISCIR2IXQ, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.905881196 (545h28m40.776964232s) [noise=true , address=145.79.227.41:4087 , piece_id=GB226GWOX7TC23BAQTUJN6QU2VO7QIONZ5VSJI4NY5ESARCH7WYA, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.908557673 (545h28m40.779640708s) [noise=true , address=145.79.227.9:4365 , piece_id=NAU2HWKLTB7NHLNBGBMHFLTL6S2BH4OPLO6FWWELH2DSPZ6MEZNQ, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.909631207 (545h28m40.780714253s) [noise=true , address=145.79.227.29:4942 , piece_id=ZMNDBC6VLZEBJ2WJ66EX3ML7VHFW7IQLPQTU5EMWIKVDEHIYKVTA, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.910935481 (545h28m40.782018516s) [noise=true , address=145.79.227.45:4229 , piece_id=FZ7D676G2K3DWTMANEAICJU7ECJZWRDLTNCFFKTDFEFGT7ADDX7A, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.913792060 (545h28m40.784875095s) [noise=true , address=145.79.227.17:4971 , piece_id=TAUKFPKSMBYPJM73TXEJJHMYM7ADJW57ZQF4V2ALOU3TJOBVHH4A, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.917421171 (545h28m40.788504216s) [noise=true , address=145.79.227.51:4409 , piece_id=TC6HJ3S7VM6OA6BEXZGM6V43WT7VY7ZP7XTM2ZUIKWAVPCXCPV7Q, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.934456098 (545h28m40.805539133s) [noise=true , address=145.79.227.3:4151 , piece_id=Z4AZOB7YRU54ALQIXNHV7K5YXBRQ2VOHPYWAYI2PVK4FZG2VCHAQ, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:41.081958449 (545h28m40.953041494s) [noise=true , address=77.90.31.19:4844 , piece_id=F3LZ7EJOCTGNF4GJBX26AGC4QB4WQ6QHSSXJ6AOQWBXRMZSL34ZA, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:41.131461521 (545h28m41.002544576s) [noise=true , address=45.91.116.9:4261 , piece_id=MMTDQLCP43W5LDA6BGDPVRBQ4E2VQ7GBEPDYOUPXFCDPX3ORRPYA, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:41.131488533 (545h28m41.002571588s) [seg_pos={0 0}, upload=478331]: Segment reached optimal threshold of 13 pieces. Dec 28 08:57:41.131771992 (545h28m41.002855038s) [noise=true , address=45.91.116.11:4119 , piece_id=5P3HKX2KJRS4GYU2GYWJ5FUAXEWVOB5EIWM3RND6MY3NOMPIBFXA, seg_pos={0 0}, upload=478331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 28 08:57:41.131782632 (545h28m41.002865677s) [noise=true , address=45.91.116.11:4119 , piece_id=5P3HKX2KJRS4GYU2GYWJ5FUAXEWVOB5EIWM3RND6MY3NOMPIBFXA, seg_pos={0 0}, upload=478331]: Long tail cancellation due to optimal threshold Dec 28 08:57:41.131968947 (545h28m41.003052012s) [noise=true , address=45.91.116.7:4010 , piece_id=PICVBYU3LMFEJ2UTHVQVHMQQB3EYZ25XXAGM6DELW5DLDJ7CHJOQ, seg_pos={0 0}, upload=478331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:44598->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:44598->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44598->45.91.116.7:4010: use of closed network connection Dec 28 08:57:41.132006558 (545h28m41.003089623s) [noise=true , address=45.91.116.7:4010 , piece_id=PICVBYU3LMFEJ2UTHVQVHMQQB3EYZ25XXAGM6DELW5DLDJ7CHJOQ, seg_pos={0 0}, upload=478331]: Long tail cancellation due to optimal threshold Dec 28 08:57:41.132100998 (545h28m41.003184034s) [noise=true , address=45.91.116.5:4297 , piece_id=NQYN7AQNQNF7Z4TJJYUCXKPVVC5I3JXXPPKA3H4V2FJDBL2GBG2A, seg_pos={0 0}, upload=478331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 28 08:57:41.132113924 (545h28m41.003196959s) [noise=true , address=45.91.116.5:4297 , piece_id=NQYN7AQNQNF7Z4TJJYUCXKPVVC5I3JXXPPKA3H4V2FJDBL2GBG2A, seg_pos={0 0}, upload=478331]: Long tail cancellation due to optimal threshold Dec 28 08:57:41.132140273 (545h28m41.003223318s) [seg_pos={0 0}, upload=478331]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 08:57:41.132230986 (545h28m41.003314031s) [upload=478331]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 08:57:41.132238360 (545h28m41.003321405s) [upload=478331]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 08:57:41.428285145 (545h28m41.29936818s) [upload=478331]: Done uploading segments... Dec 28 08:57:41.428349467 (545h28m41.299432513s) [upload=478331]: Upload finished. err: <nil> Duration: 5.686289286s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [X.XX.......XX.X....XXXX.XX....X....X.XX..XX..X........X.X...] duration: 4.089233384s trace: https://jaeger.dev.storj.io/trace/46f1c877a37f9edc FAIL (2025-12-28 09:00:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.089261266s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.X.XX..X..XX....X...X.XX.XX.......X..X......XXXX.....XX.X...] duration: 5.929243283s trace: https://jaeger.dev.storj.io/trace/4472a4ab736f5ac FAIL (2025-12-28 08:59:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 08:57:38.644991590 (545h28m38.516074635s) [upload=478331]: Starting upload Dec 28 08:57:38.645074659 (545h28m38.516157704s) [upload=478331]: Done starting upload Dec 28 08:57:38.645239703 (545h28m38.516322738s) [upload=478331]: Uploading segments... Dec 28 08:57:38.645381684 (545h28m38.516464729s) [upload=478331]: Got next segment. Inline: false Dec 28 08:57:38.645426229 (545h28m38.516509264s) [upload=478331]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 08:57:38.645465563 (545h28m38.516548608s) [upload=478331]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 08:57:38.894465663 (545h28m38.765548718s) [seg_pos={0 0}, upload=478331]: Begin upload segment... Dec 28 08:57:38.894532079 (545h28m38.765615114s) [seg_pos={0 0}, upload=478331]: Done begin upload segment. Dec 28 08:57:38.894574179 (545h28m38.765657214s) [noise=true , address=145.79.227.50:4373 , piece_id=QT3FSYZYIGJIPOHPO2L3VHG2AXPPNV4KVRJ5OPIDJ7T43RKMYKKQ, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.894676474 (545h28m38.765759519s) [noise=true , address=145.79.227.45:4229 , piece_id=FZ7D676G2K3DWTMANEAICJU7ECJZWRDLTNCFFKTDFEFGT7ADDX7A, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.895532635 (545h28m38.76661567s) [noise=true , address=77.90.31.19:4844 , piece_id=F3LZ7EJOCTGNF4GJBX26AGC4QB4WQ6QHSSXJ6AOQWBXRMZSL34ZA, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.895644868 (545h28m38.766727913s) [noise=true , address=45.91.116.9:4261 , piece_id=MMTDQLCP43W5LDA6BGDPVRBQ4E2VQ7GBEPDYOUPXFCDPX3ORRPYA, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.896029460 (545h28m38.767112495s) [noise=true , address=45.91.116.7:4010 , piece_id=PICVBYU3LMFEJ2UTHVQVHMQQB3EYZ25XXAGM6DELW5DLDJ7CHJOQ, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.896322279 (545h28m38.767405314s) [noise=true , address=145.79.227.3:4151 , piece_id=Z4AZOB7YRU54ALQIXNHV7K5YXBRQ2VOHPYWAYI2PVK4FZG2VCHAQ, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.896682224 (545h28m38.767765269s) [noise=true , address=145.79.227.51:4409 , piece_id=TC6HJ3S7VM6OA6BEXZGM6V43WT7VY7ZP7XTM2ZUIKWAVPCXCPV7Q, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.896720056 (545h28m38.767803101s) [noise=true , address=145.79.227.29:4942 , piece_id=ZMNDBC6VLZEBJ2WJ66EX3ML7VHFW7IQLPQTU5EMWIKVDEHIYKVTA, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.897026940 (545h28m38.768109985s) [noise=true , address=145.79.227.27:4691 , piece_id=RYCQQN4SLZGLJ4NDSLPXZGDSAXCEDC3PYMQ7KW7NJKFW22COSCPA, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.897134114 (545h28m38.768217149s) [noise=true , address=145.79.227.11:4731 , piece_id=KIWK6CLD5OHQ7XEOHPURMMPIP6WCHUNDVQR7KYTO73D5JYM5K37Q, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.897431581 (545h28m38.768514626s) [noise=true , address=145.79.227.39:4013 , piece_id=MCN5VHIKQTFS2WY6NENYVETCIQ5COTMHDMW4BO3FS56ISCIR2IXQ, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.897624919 (545h28m38.768707954s) [noise=true , address=45.91.116.11:4119 , piece_id=5P3HKX2KJRS4GYU2GYWJ5FUAXEWVOB5EIWM3RND6MY3NOMPIBFXA, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.897872700 (545h28m38.768955745s) [noise=true , address=145.79.227.41:4087 , piece_id=GB226GWOX7TC23BAQTUJN6QU2VO7QIONZ5VSJI4NY5ESARCH7WYA, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.898120582 (545h28m38.769203617s) [noise=true , address=45.91.116.5:4297 , piece_id=NQYN7AQNQNF7Z4TJJYUCXKPVVC5I3JXXPPKA3H4V2FJDBL2GBG2A, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.898505405 (545h28m38.76958844s) [noise=true , address=145.79.227.9:4365 , piece_id=NAU2HWKLTB7NHLNBGBMHFLTL6S2BH4OPLO6FWWELH2DSPZ6MEZNQ, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.898901520 (545h28m38.769984555s) [noise=true , address=145.79.227.17:4971 , piece_id=TAUKFPKSMBYPJM73TXEJJHMYM7ADJW57ZQF4V2ALOU3TJOBVHH4A, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:40.609896701 (545h28m40.480979736s) [upload=478331]: Next returned nil segment Dec 28 08:57:40.609912210 (545h28m40.480995245s) [upload=478331]: Waiting for error group managing segments... Dec 28 08:57:40.894791250 (545h28m40.765874295s) [noise=true , address=145.79.227.27:4691 , piece_id=RYCQQN4SLZGLJ4NDSLPXZGDSAXCEDC3PYMQ7KW7NJKFW22COSCPA, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.901466881 (545h28m40.772549916s) [noise=true , address=145.79.227.50:4373 , piece_id=QT3FSYZYIGJIPOHPO2L3VHG2AXPPNV4KVRJ5OPIDJ7T43RKMYKKQ, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.902367305 (545h28m40.77345035s) [noise=true , address=145.79.227.11:4731 , piece_id=KIWK6CLD5OHQ7XEOHPURMMPIP6WCHUNDVQR7KYTO73D5JYM5K37Q, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.903421053 (545h28m40.774504088s) [noise=true , address=145.79.227.39:4013 , piece_id=MCN5VHIKQTFS2WY6NENYVETCIQ5COTMHDMW4BO3FS56ISCIR2IXQ, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.905881196 (545h28m40.776964232s) [noise=true , address=145.79.227.41:4087 , piece_id=GB226GWOX7TC23BAQTUJN6QU2VO7QIONZ5VSJI4NY5ESARCH7WYA, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.908557673 (545h28m40.779640708s) [noise=true , address=145.79.227.9:4365 , piece_id=NAU2HWKLTB7NHLNBGBMHFLTL6S2BH4OPLO6FWWELH2DSPZ6MEZNQ, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.909631207 (545h28m40.780714253s) [noise=true , address=145.79.227.29:4942 , piece_id=ZMNDBC6VLZEBJ2WJ66EX3ML7VHFW7IQLPQTU5EMWIKVDEHIYKVTA, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.910935481 (545h28m40.782018516s) [noise=true , address=145.79.227.45:4229 , piece_id=FZ7D676G2K3DWTMANEAICJU7ECJZWRDLTNCFFKTDFEFGT7ADDX7A, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.913792060 (545h28m40.784875095s) [noise=true , address=145.79.227.17:4971 , piece_id=TAUKFPKSMBYPJM73TXEJJHMYM7ADJW57ZQF4V2ALOU3TJOBVHH4A, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.917421171 (545h28m40.788504216s) [noise=true , address=145.79.227.51:4409 , piece_id=TC6HJ3S7VM6OA6BEXZGM6V43WT7VY7ZP7XTM2ZUIKWAVPCXCPV7Q, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.934456098 (545h28m40.805539133s) [noise=true , address=145.79.227.3:4151 , piece_id=Z4AZOB7YRU54ALQIXNHV7K5YXBRQ2VOHPYWAYI2PVK4FZG2VCHAQ, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:41.081958449 (545h28m40.953041494s) [noise=true , address=77.90.31.19:4844 , piece_id=F3LZ7EJOCTGNF4GJBX26AGC4QB4WQ6QHSSXJ6AOQWBXRMZSL34ZA, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:41.131461521 (545h28m41.002544576s) [noise=true , address=45.91.116.9:4261 , piece_id=MMTDQLCP43W5LDA6BGDPVRBQ4E2VQ7GBEPDYOUPXFCDPX3ORRPYA, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:41.131488533 (545h28m41.002571588s) [seg_pos={0 0}, upload=478331]: Segment reached optimal threshold of 13 pieces. Dec 28 08:57:41.131771992 (545h28m41.002855038s) [noise=true , address=45.91.116.11:4119 , piece_id=5P3HKX2KJRS4GYU2GYWJ5FUAXEWVOB5EIWM3RND6MY3NOMPIBFXA, seg_pos={0 0}, upload=478331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 28 08:57:41.131782632 (545h28m41.002865677s) [noise=true , address=45.91.116.11:4119 , piece_id=5P3HKX2KJRS4GYU2GYWJ5FUAXEWVOB5EIWM3RND6MY3NOMPIBFXA, seg_pos={0 0}, upload=478331]: Long tail cancellation due to optimal threshold Dec 28 08:57:41.131968947 (545h28m41.003052012s) [noise=true , address=45.91.116.7:4010 , piece_id=PICVBYU3LMFEJ2UTHVQVHMQQB3EYZ25XXAGM6DELW5DLDJ7CHJOQ, seg_pos={0 0}, upload=478331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:44598->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:44598->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44598->45.91.116.7:4010: use of closed network connection Dec 28 08:57:41.132006558 (545h28m41.003089623s) [noise=true , address=45.91.116.7:4010 , piece_id=PICVBYU3LMFEJ2UTHVQVHMQQB3EYZ25XXAGM6DELW5DLDJ7CHJOQ, seg_pos={0 0}, upload=478331]: Long tail cancellation due to optimal threshold Dec 28 08:57:41.132100998 (545h28m41.003184034s) [noise=true , address=45.91.116.5:4297 , piece_id=NQYN7AQNQNF7Z4TJJYUCXKPVVC5I3JXXPPKA3H4V2FJDBL2GBG2A, seg_pos={0 0}, upload=478331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 28 08:57:41.132113924 (545h28m41.003196959s) [noise=true , address=45.91.116.5:4297 , piece_id=NQYN7AQNQNF7Z4TJJYUCXKPVVC5I3JXXPPKA3H4V2FJDBL2GBG2A, seg_pos={0 0}, upload=478331]: Long tail cancellation due to optimal threshold Dec 28 08:57:41.132140273 (545h28m41.003223318s) [seg_pos={0 0}, upload=478331]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 08:57:41.132230986 (545h28m41.003314031s) [upload=478331]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 08:57:41.132238360 (545h28m41.003321405s) [upload=478331]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 08:57:41.428285145 (545h28m41.29936818s) [upload=478331]: Done uploading segments... Dec 28 08:57:41.428349467 (545h28m41.299432513s) [upload=478331]: Upload finished. err: <nil> Duration: 5.929276938s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X.X.XX..X..XX....X...X.XX.XX.......X..X......XXXX.....XX.X..] duration: 5.707867705s trace: https://jaeger.dev.storj.io/trace/4472a4ab736f5ac FAIL (2025-12-28 08:59:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 08:57:38.644991590 (545h28m38.516074635s) [upload=478331]: Starting upload Dec 28 08:57:38.645074659 (545h28m38.516157704s) [upload=478331]: Done starting upload Dec 28 08:57:38.645239703 (545h28m38.516322738s) [upload=478331]: Uploading segments... Dec 28 08:57:38.645381684 (545h28m38.516464729s) [upload=478331]: Got next segment. Inline: false Dec 28 08:57:38.645426229 (545h28m38.516509264s) [upload=478331]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 08:57:38.645465563 (545h28m38.516548608s) [upload=478331]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 08:57:38.894465663 (545h28m38.765548718s) [seg_pos={0 0}, upload=478331]: Begin upload segment... Dec 28 08:57:38.894532079 (545h28m38.765615114s) [seg_pos={0 0}, upload=478331]: Done begin upload segment. Dec 28 08:57:38.894574179 (545h28m38.765657214s) [noise=true , address=145.79.227.50:4373 , piece_id=QT3FSYZYIGJIPOHPO2L3VHG2AXPPNV4KVRJ5OPIDJ7T43RKMYKKQ, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.894676474 (545h28m38.765759519s) [noise=true , address=145.79.227.45:4229 , piece_id=FZ7D676G2K3DWTMANEAICJU7ECJZWRDLTNCFFKTDFEFGT7ADDX7A, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.895532635 (545h28m38.76661567s) [noise=true , address=77.90.31.19:4844 , piece_id=F3LZ7EJOCTGNF4GJBX26AGC4QB4WQ6QHSSXJ6AOQWBXRMZSL34ZA, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.895644868 (545h28m38.766727913s) [noise=true , address=45.91.116.9:4261 , piece_id=MMTDQLCP43W5LDA6BGDPVRBQ4E2VQ7GBEPDYOUPXFCDPX3ORRPYA, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.896029460 (545h28m38.767112495s) [noise=true , address=45.91.116.7:4010 , piece_id=PICVBYU3LMFEJ2UTHVQVHMQQB3EYZ25XXAGM6DELW5DLDJ7CHJOQ, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.896322279 (545h28m38.767405314s) [noise=true , address=145.79.227.3:4151 , piece_id=Z4AZOB7YRU54ALQIXNHV7K5YXBRQ2VOHPYWAYI2PVK4FZG2VCHAQ, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.896682224 (545h28m38.767765269s) [noise=true , address=145.79.227.51:4409 , piece_id=TC6HJ3S7VM6OA6BEXZGM6V43WT7VY7ZP7XTM2ZUIKWAVPCXCPV7Q, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.896720056 (545h28m38.767803101s) [noise=true , address=145.79.227.29:4942 , piece_id=ZMNDBC6VLZEBJ2WJ66EX3ML7VHFW7IQLPQTU5EMWIKVDEHIYKVTA, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.897026940 (545h28m38.768109985s) [noise=true , address=145.79.227.27:4691 , piece_id=RYCQQN4SLZGLJ4NDSLPXZGDSAXCEDC3PYMQ7KW7NJKFW22COSCPA, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.897134114 (545h28m38.768217149s) [noise=true , address=145.79.227.11:4731 , piece_id=KIWK6CLD5OHQ7XEOHPURMMPIP6WCHUNDVQR7KYTO73D5JYM5K37Q, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.897431581 (545h28m38.768514626s) [noise=true , address=145.79.227.39:4013 , piece_id=MCN5VHIKQTFS2WY6NENYVETCIQ5COTMHDMW4BO3FS56ISCIR2IXQ, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.897624919 (545h28m38.768707954s) [noise=true , address=45.91.116.11:4119 , piece_id=5P3HKX2KJRS4GYU2GYWJ5FUAXEWVOB5EIWM3RND6MY3NOMPIBFXA, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.897872700 (545h28m38.768955745s) [noise=true , address=145.79.227.41:4087 , piece_id=GB226GWOX7TC23BAQTUJN6QU2VO7QIONZ5VSJI4NY5ESARCH7WYA, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.898120582 (545h28m38.769203617s) [noise=true , address=45.91.116.5:4297 , piece_id=NQYN7AQNQNF7Z4TJJYUCXKPVVC5I3JXXPPKA3H4V2FJDBL2GBG2A, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.898505405 (545h28m38.76958844s) [noise=true , address=145.79.227.9:4365 , piece_id=NAU2HWKLTB7NHLNBGBMHFLTL6S2BH4OPLO6FWWELH2DSPZ6MEZNQ, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.898901520 (545h28m38.769984555s) [noise=true , address=145.79.227.17:4971 , piece_id=TAUKFPKSMBYPJM73TXEJJHMYM7ADJW57ZQF4V2ALOU3TJOBVHH4A, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:40.609896701 (545h28m40.480979736s) [upload=478331]: Next returned nil segment Dec 28 08:57:40.609912210 (545h28m40.480995245s) [upload=478331]: Waiting for error group managing segments... Dec 28 08:57:40.894791250 (545h28m40.765874295s) [noise=true , address=145.79.227.27:4691 , piece_id=RYCQQN4SLZGLJ4NDSLPXZGDSAXCEDC3PYMQ7KW7NJKFW22COSCPA, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.901466881 (545h28m40.772549916s) [noise=true , address=145.79.227.50:4373 , piece_id=QT3FSYZYIGJIPOHPO2L3VHG2AXPPNV4KVRJ5OPIDJ7T43RKMYKKQ, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.902367305 (545h28m40.77345035s) [noise=true , address=145.79.227.11:4731 , piece_id=KIWK6CLD5OHQ7XEOHPURMMPIP6WCHUNDVQR7KYTO73D5JYM5K37Q, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.903421053 (545h28m40.774504088s) [noise=true , address=145.79.227.39:4013 , piece_id=MCN5VHIKQTFS2WY6NENYVETCIQ5COTMHDMW4BO3FS56ISCIR2IXQ, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.905881196 (545h28m40.776964232s) [noise=true , address=145.79.227.41:4087 , piece_id=GB226GWOX7TC23BAQTUJN6QU2VO7QIONZ5VSJI4NY5ESARCH7WYA, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.908557673 (545h28m40.779640708s) [noise=true , address=145.79.227.9:4365 , piece_id=NAU2HWKLTB7NHLNBGBMHFLTL6S2BH4OPLO6FWWELH2DSPZ6MEZNQ, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.909631207 (545h28m40.780714253s) [noise=true , address=145.79.227.29:4942 , piece_id=ZMNDBC6VLZEBJ2WJ66EX3ML7VHFW7IQLPQTU5EMWIKVDEHIYKVTA, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.910935481 (545h28m40.782018516s) [noise=true , address=145.79.227.45:4229 , piece_id=FZ7D676G2K3DWTMANEAICJU7ECJZWRDLTNCFFKTDFEFGT7ADDX7A, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.913792060 (545h28m40.784875095s) [noise=true , address=145.79.227.17:4971 , piece_id=TAUKFPKSMBYPJM73TXEJJHMYM7ADJW57ZQF4V2ALOU3TJOBVHH4A, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.917421171 (545h28m40.788504216s) [noise=true , address=145.79.227.51:4409 , piece_id=TC6HJ3S7VM6OA6BEXZGM6V43WT7VY7ZP7XTM2ZUIKWAVPCXCPV7Q, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.934456098 (545h28m40.805539133s) [noise=true , address=145.79.227.3:4151 , piece_id=Z4AZOB7YRU54ALQIXNHV7K5YXBRQ2VOHPYWAYI2PVK4FZG2VCHAQ, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:41.081958449 (545h28m40.953041494s) [noise=true , address=77.90.31.19:4844 , piece_id=F3LZ7EJOCTGNF4GJBX26AGC4QB4WQ6QHSSXJ6AOQWBXRMZSL34ZA, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:41.131461521 (545h28m41.002544576s) [noise=true , address=45.91.116.9:4261 , piece_id=MMTDQLCP43W5LDA6BGDPVRBQ4E2VQ7GBEPDYOUPXFCDPX3ORRPYA, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:41.131488533 (545h28m41.002571588s) [seg_pos={0 0}, upload=478331]: Segment reached optimal threshold of 13 pieces. Dec 28 08:57:41.131771992 (545h28m41.002855038s) [noise=true , address=45.91.116.11:4119 , piece_id=5P3HKX2KJRS4GYU2GYWJ5FUAXEWVOB5EIWM3RND6MY3NOMPIBFXA, seg_pos={0 0}, upload=478331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 28 08:57:41.131782632 (545h28m41.002865677s) [noise=true , address=45.91.116.11:4119 , piece_id=5P3HKX2KJRS4GYU2GYWJ5FUAXEWVOB5EIWM3RND6MY3NOMPIBFXA, seg_pos={0 0}, upload=478331]: Long tail cancellation due to optimal threshold Dec 28 08:57:41.131968947 (545h28m41.003052012s) [noise=true , address=45.91.116.7:4010 , piece_id=PICVBYU3LMFEJ2UTHVQVHMQQB3EYZ25XXAGM6DELW5DLDJ7CHJOQ, seg_pos={0 0}, upload=478331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:44598->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:44598->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44598->45.91.116.7:4010: use of closed network connection Dec 28 08:57:41.132006558 (545h28m41.003089623s) [noise=true , address=45.91.116.7:4010 , piece_id=PICVBYU3LMFEJ2UTHVQVHMQQB3EYZ25XXAGM6DELW5DLDJ7CHJOQ, seg_pos={0 0}, upload=478331]: Long tail cancellation due to optimal threshold Dec 28 08:57:41.132100998 (545h28m41.003184034s) [noise=true , address=45.91.116.5:4297 , piece_id=NQYN7AQNQNF7Z4TJJYUCXKPVVC5I3JXXPPKA3H4V2FJDBL2GBG2A, seg_pos={0 0}, upload=478331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 28 08:57:41.132113924 (545h28m41.003196959s) [noise=true , address=45.91.116.5:4297 , piece_id=NQYN7AQNQNF7Z4TJJYUCXKPVVC5I3JXXPPKA3H4V2FJDBL2GBG2A, seg_pos={0 0}, upload=478331]: Long tail cancellation due to optimal threshold Dec 28 08:57:41.132140273 (545h28m41.003223318s) [seg_pos={0 0}, upload=478331]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 08:57:41.132230986 (545h28m41.003314031s) [upload=478331]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 08:57:41.132238360 (545h28m41.003321405s) [upload=478331]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 08:57:41.428285145 (545h28m41.29936818s) [upload=478331]: Done uploading segments... Dec 28 08:57:41.428349467 (545h28m41.299432513s) [upload=478331]: Upload finished. err: <nil> Duration: 5.707898153s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [XX.XX.......XX.X....XXXX.XX....X....X.XX..XX..X........X.X..] duration: 4.619959388s trace: https://jaeger.dev.storj.io/trace/46f1c877a37f9edc FAIL (2025-12-28 08:59:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.620032857s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [.XX.XX.......XX.X....XXXX.XX....X....X.XX..XX..X........X.X.] duration: 3.589719342s trace: https://jaeger.dev.storj.io/trace/46f1c877a37f9edc FAIL (2025-12-28 08:58:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.589786068s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [XX.X.XX..X..XX....X...X.XX.XX.......X..X......XXXX.....XX.X.] duration: 9.933835473s trace: https://jaeger.dev.storj.io/trace/4472a4ab736f5ac FAIL (2025-12-28 08:58:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 08:57:38.644991590 (545h28m38.516074635s) [upload=478331]: Starting upload Dec 28 08:57:38.645074659 (545h28m38.516157704s) [upload=478331]: Done starting upload Dec 28 08:57:38.645239703 (545h28m38.516322738s) [upload=478331]: Uploading segments... Dec 28 08:57:38.645381684 (545h28m38.516464729s) [upload=478331]: Got next segment. Inline: false Dec 28 08:57:38.645426229 (545h28m38.516509264s) [upload=478331]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 08:57:38.645465563 (545h28m38.516548608s) [upload=478331]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 08:57:38.894465663 (545h28m38.765548718s) [seg_pos={0 0}, upload=478331]: Begin upload segment... Dec 28 08:57:38.894532079 (545h28m38.765615114s) [seg_pos={0 0}, upload=478331]: Done begin upload segment. Dec 28 08:57:38.894574179 (545h28m38.765657214s) [noise=true , address=145.79.227.50:4373 , piece_id=QT3FSYZYIGJIPOHPO2L3VHG2AXPPNV4KVRJ5OPIDJ7T43RKMYKKQ, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.894676474 (545h28m38.765759519s) [noise=true , address=145.79.227.45:4229 , piece_id=FZ7D676G2K3DWTMANEAICJU7ECJZWRDLTNCFFKTDFEFGT7ADDX7A, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.895532635 (545h28m38.76661567s) [noise=true , address=77.90.31.19:4844 , piece_id=F3LZ7EJOCTGNF4GJBX26AGC4QB4WQ6QHSSXJ6AOQWBXRMZSL34ZA, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.895644868 (545h28m38.766727913s) [noise=true , address=45.91.116.9:4261 , piece_id=MMTDQLCP43W5LDA6BGDPVRBQ4E2VQ7GBEPDYOUPXFCDPX3ORRPYA, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.896029460 (545h28m38.767112495s) [noise=true , address=45.91.116.7:4010 , piece_id=PICVBYU3LMFEJ2UTHVQVHMQQB3EYZ25XXAGM6DELW5DLDJ7CHJOQ, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.896322279 (545h28m38.767405314s) [noise=true , address=145.79.227.3:4151 , piece_id=Z4AZOB7YRU54ALQIXNHV7K5YXBRQ2VOHPYWAYI2PVK4FZG2VCHAQ, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.896682224 (545h28m38.767765269s) [noise=true , address=145.79.227.51:4409 , piece_id=TC6HJ3S7VM6OA6BEXZGM6V43WT7VY7ZP7XTM2ZUIKWAVPCXCPV7Q, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.896720056 (545h28m38.767803101s) [noise=true , address=145.79.227.29:4942 , piece_id=ZMNDBC6VLZEBJ2WJ66EX3ML7VHFW7IQLPQTU5EMWIKVDEHIYKVTA, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.897026940 (545h28m38.768109985s) [noise=true , address=145.79.227.27:4691 , piece_id=RYCQQN4SLZGLJ4NDSLPXZGDSAXCEDC3PYMQ7KW7NJKFW22COSCPA, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.897134114 (545h28m38.768217149s) [noise=true , address=145.79.227.11:4731 , piece_id=KIWK6CLD5OHQ7XEOHPURMMPIP6WCHUNDVQR7KYTO73D5JYM5K37Q, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.897431581 (545h28m38.768514626s) [noise=true , address=145.79.227.39:4013 , piece_id=MCN5VHIKQTFS2WY6NENYVETCIQ5COTMHDMW4BO3FS56ISCIR2IXQ, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.897624919 (545h28m38.768707954s) [noise=true , address=45.91.116.11:4119 , piece_id=5P3HKX2KJRS4GYU2GYWJ5FUAXEWVOB5EIWM3RND6MY3NOMPIBFXA, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.897872700 (545h28m38.768955745s) [noise=true , address=145.79.227.41:4087 , piece_id=GB226GWOX7TC23BAQTUJN6QU2VO7QIONZ5VSJI4NY5ESARCH7WYA, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.898120582 (545h28m38.769203617s) [noise=true , address=45.91.116.5:4297 , piece_id=NQYN7AQNQNF7Z4TJJYUCXKPVVC5I3JXXPPKA3H4V2FJDBL2GBG2A, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.898505405 (545h28m38.76958844s) [noise=true , address=145.79.227.9:4365 , piece_id=NAU2HWKLTB7NHLNBGBMHFLTL6S2BH4OPLO6FWWELH2DSPZ6MEZNQ, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.898901520 (545h28m38.769984555s) [noise=true , address=145.79.227.17:4971 , piece_id=TAUKFPKSMBYPJM73TXEJJHMYM7ADJW57ZQF4V2ALOU3TJOBVHH4A, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:40.609896701 (545h28m40.480979736s) [upload=478331]: Next returned nil segment Dec 28 08:57:40.609912210 (545h28m40.480995245s) [upload=478331]: Waiting for error group managing segments... Dec 28 08:57:40.894791250 (545h28m40.765874295s) [noise=true , address=145.79.227.27:4691 , piece_id=RYCQQN4SLZGLJ4NDSLPXZGDSAXCEDC3PYMQ7KW7NJKFW22COSCPA, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.901466881 (545h28m40.772549916s) [noise=true , address=145.79.227.50:4373 , piece_id=QT3FSYZYIGJIPOHPO2L3VHG2AXPPNV4KVRJ5OPIDJ7T43RKMYKKQ, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.902367305 (545h28m40.77345035s) [noise=true , address=145.79.227.11:4731 , piece_id=KIWK6CLD5OHQ7XEOHPURMMPIP6WCHUNDVQR7KYTO73D5JYM5K37Q, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.903421053 (545h28m40.774504088s) [noise=true , address=145.79.227.39:4013 , piece_id=MCN5VHIKQTFS2WY6NENYVETCIQ5COTMHDMW4BO3FS56ISCIR2IXQ, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.905881196 (545h28m40.776964232s) [noise=true , address=145.79.227.41:4087 , piece_id=GB226GWOX7TC23BAQTUJN6QU2VO7QIONZ5VSJI4NY5ESARCH7WYA, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.908557673 (545h28m40.779640708s) [noise=true , address=145.79.227.9:4365 , piece_id=NAU2HWKLTB7NHLNBGBMHFLTL6S2BH4OPLO6FWWELH2DSPZ6MEZNQ, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.909631207 (545h28m40.780714253s) [noise=true , address=145.79.227.29:4942 , piece_id=ZMNDBC6VLZEBJ2WJ66EX3ML7VHFW7IQLPQTU5EMWIKVDEHIYKVTA, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.910935481 (545h28m40.782018516s) [noise=true , address=145.79.227.45:4229 , piece_id=FZ7D676G2K3DWTMANEAICJU7ECJZWRDLTNCFFKTDFEFGT7ADDX7A, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.913792060 (545h28m40.784875095s) [noise=true , address=145.79.227.17:4971 , piece_id=TAUKFPKSMBYPJM73TXEJJHMYM7ADJW57ZQF4V2ALOU3TJOBVHH4A, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.917421171 (545h28m40.788504216s) [noise=true , address=145.79.227.51:4409 , piece_id=TC6HJ3S7VM6OA6BEXZGM6V43WT7VY7ZP7XTM2ZUIKWAVPCXCPV7Q, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.934456098 (545h28m40.805539133s) [noise=true , address=145.79.227.3:4151 , piece_id=Z4AZOB7YRU54ALQIXNHV7K5YXBRQ2VOHPYWAYI2PVK4FZG2VCHAQ, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:41.081958449 (545h28m40.953041494s) [noise=true , address=77.90.31.19:4844 , piece_id=F3LZ7EJOCTGNF4GJBX26AGC4QB4WQ6QHSSXJ6AOQWBXRMZSL34ZA, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:41.131461521 (545h28m41.002544576s) [noise=true , address=45.91.116.9:4261 , piece_id=MMTDQLCP43W5LDA6BGDPVRBQ4E2VQ7GBEPDYOUPXFCDPX3ORRPYA, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:41.131488533 (545h28m41.002571588s) [seg_pos={0 0}, upload=478331]: Segment reached optimal threshold of 13 pieces. Dec 28 08:57:41.131771992 (545h28m41.002855038s) [noise=true , address=45.91.116.11:4119 , piece_id=5P3HKX2KJRS4GYU2GYWJ5FUAXEWVOB5EIWM3RND6MY3NOMPIBFXA, seg_pos={0 0}, upload=478331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 28 08:57:41.131782632 (545h28m41.002865677s) [noise=true , address=45.91.116.11:4119 , piece_id=5P3HKX2KJRS4GYU2GYWJ5FUAXEWVOB5EIWM3RND6MY3NOMPIBFXA, seg_pos={0 0}, upload=478331]: Long tail cancellation due to optimal threshold Dec 28 08:57:41.131968947 (545h28m41.003052012s) [noise=true , address=45.91.116.7:4010 , piece_id=PICVBYU3LMFEJ2UTHVQVHMQQB3EYZ25XXAGM6DELW5DLDJ7CHJOQ, seg_pos={0 0}, upload=478331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:44598->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:44598->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44598->45.91.116.7:4010: use of closed network connection Dec 28 08:57:41.132006558 (545h28m41.003089623s) [noise=true , address=45.91.116.7:4010 , piece_id=PICVBYU3LMFEJ2UTHVQVHMQQB3EYZ25XXAGM6DELW5DLDJ7CHJOQ, seg_pos={0 0}, upload=478331]: Long tail cancellation due to optimal threshold Dec 28 08:57:41.132100998 (545h28m41.003184034s) [noise=true , address=45.91.116.5:4297 , piece_id=NQYN7AQNQNF7Z4TJJYUCXKPVVC5I3JXXPPKA3H4V2FJDBL2GBG2A, seg_pos={0 0}, upload=478331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 28 08:57:41.132113924 (545h28m41.003196959s) [noise=true , address=45.91.116.5:4297 , piece_id=NQYN7AQNQNF7Z4TJJYUCXKPVVC5I3JXXPPKA3H4V2FJDBL2GBG2A, seg_pos={0 0}, upload=478331]: Long tail cancellation due to optimal threshold Dec 28 08:57:41.132140273 (545h28m41.003223318s) [seg_pos={0 0}, upload=478331]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 08:57:41.132230986 (545h28m41.003314031s) [upload=478331]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 08:57:41.132238360 (545h28m41.003321405s) [upload=478331]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 08:57:41.428285145 (545h28m41.29936818s) [upload=478331]: Done uploading segments... Dec 28 08:57:41.428349467 (545h28m41.299432513s) [upload=478331]: Upload finished. err: <nil> Duration: 9.933862205s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.XX.X.XX..X..XX....X...X.XX.XX.......X..X......XXXX.....XX.X] duration: 30.002796415s trace: https://jaeger.dev.storj.io/trace/4472a4ab736f5ac FAIL (2025-12-28 08:58:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 08:57:38.644991590 (545h28m38.516074635s) [upload=478331]: Starting upload Dec 28 08:57:38.645074659 (545h28m38.516157704s) [upload=478331]: Done starting upload Dec 28 08:57:38.645239703 (545h28m38.516322738s) [upload=478331]: Uploading segments... Dec 28 08:57:38.645381684 (545h28m38.516464729s) [upload=478331]: Got next segment. Inline: false Dec 28 08:57:38.645426229 (545h28m38.516509264s) [upload=478331]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 08:57:38.645465563 (545h28m38.516548608s) [upload=478331]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 08:57:38.894465663 (545h28m38.765548718s) [seg_pos={0 0}, upload=478331]: Begin upload segment... Dec 28 08:57:38.894532079 (545h28m38.765615114s) [seg_pos={0 0}, upload=478331]: Done begin upload segment. Dec 28 08:57:38.894574179 (545h28m38.765657214s) [noise=true , address=145.79.227.50:4373 , piece_id=QT3FSYZYIGJIPOHPO2L3VHG2AXPPNV4KVRJ5OPIDJ7T43RKMYKKQ, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.894676474 (545h28m38.765759519s) [noise=true , address=145.79.227.45:4229 , piece_id=FZ7D676G2K3DWTMANEAICJU7ECJZWRDLTNCFFKTDFEFGT7ADDX7A, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.895532635 (545h28m38.76661567s) [noise=true , address=77.90.31.19:4844 , piece_id=F3LZ7EJOCTGNF4GJBX26AGC4QB4WQ6QHSSXJ6AOQWBXRMZSL34ZA, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.895644868 (545h28m38.766727913s) [noise=true , address=45.91.116.9:4261 , piece_id=MMTDQLCP43W5LDA6BGDPVRBQ4E2VQ7GBEPDYOUPXFCDPX3ORRPYA, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.896029460 (545h28m38.767112495s) [noise=true , address=45.91.116.7:4010 , piece_id=PICVBYU3LMFEJ2UTHVQVHMQQB3EYZ25XXAGM6DELW5DLDJ7CHJOQ, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.896322279 (545h28m38.767405314s) [noise=true , address=145.79.227.3:4151 , piece_id=Z4AZOB7YRU54ALQIXNHV7K5YXBRQ2VOHPYWAYI2PVK4FZG2VCHAQ, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.896682224 (545h28m38.767765269s) [noise=true , address=145.79.227.51:4409 , piece_id=TC6HJ3S7VM6OA6BEXZGM6V43WT7VY7ZP7XTM2ZUIKWAVPCXCPV7Q, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.896720056 (545h28m38.767803101s) [noise=true , address=145.79.227.29:4942 , piece_id=ZMNDBC6VLZEBJ2WJ66EX3ML7VHFW7IQLPQTU5EMWIKVDEHIYKVTA, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.897026940 (545h28m38.768109985s) [noise=true , address=145.79.227.27:4691 , piece_id=RYCQQN4SLZGLJ4NDSLPXZGDSAXCEDC3PYMQ7KW7NJKFW22COSCPA, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.897134114 (545h28m38.768217149s) [noise=true , address=145.79.227.11:4731 , piece_id=KIWK6CLD5OHQ7XEOHPURMMPIP6WCHUNDVQR7KYTO73D5JYM5K37Q, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.897431581 (545h28m38.768514626s) [noise=true , address=145.79.227.39:4013 , piece_id=MCN5VHIKQTFS2WY6NENYVETCIQ5COTMHDMW4BO3FS56ISCIR2IXQ, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.897624919 (545h28m38.768707954s) [noise=true , address=45.91.116.11:4119 , piece_id=5P3HKX2KJRS4GYU2GYWJ5FUAXEWVOB5EIWM3RND6MY3NOMPIBFXA, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.897872700 (545h28m38.768955745s) [noise=true , address=145.79.227.41:4087 , piece_id=GB226GWOX7TC23BAQTUJN6QU2VO7QIONZ5VSJI4NY5ESARCH7WYA, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.898120582 (545h28m38.769203617s) [noise=true , address=45.91.116.5:4297 , piece_id=NQYN7AQNQNF7Z4TJJYUCXKPVVC5I3JXXPPKA3H4V2FJDBL2GBG2A, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.898505405 (545h28m38.76958844s) [noise=true , address=145.79.227.9:4365 , piece_id=NAU2HWKLTB7NHLNBGBMHFLTL6S2BH4OPLO6FWWELH2DSPZ6MEZNQ, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:38.898901520 (545h28m38.769984555s) [noise=true , address=145.79.227.17:4971 , piece_id=TAUKFPKSMBYPJM73TXEJJHMYM7ADJW57ZQF4V2ALOU3TJOBVHH4A, seg_pos={0 0}, upload=478331]: Uploading piece... Dec 28 08:57:40.609896701 (545h28m40.480979736s) [upload=478331]: Next returned nil segment Dec 28 08:57:40.609912210 (545h28m40.480995245s) [upload=478331]: Waiting for error group managing segments... Dec 28 08:57:40.894791250 (545h28m40.765874295s) [noise=true , address=145.79.227.27:4691 , piece_id=RYCQQN4SLZGLJ4NDSLPXZGDSAXCEDC3PYMQ7KW7NJKFW22COSCPA, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.901466881 (545h28m40.772549916s) [noise=true , address=145.79.227.50:4373 , piece_id=QT3FSYZYIGJIPOHPO2L3VHG2AXPPNV4KVRJ5OPIDJ7T43RKMYKKQ, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.902367305 (545h28m40.77345035s) [noise=true , address=145.79.227.11:4731 , piece_id=KIWK6CLD5OHQ7XEOHPURMMPIP6WCHUNDVQR7KYTO73D5JYM5K37Q, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.903421053 (545h28m40.774504088s) [noise=true , address=145.79.227.39:4013 , piece_id=MCN5VHIKQTFS2WY6NENYVETCIQ5COTMHDMW4BO3FS56ISCIR2IXQ, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.905881196 (545h28m40.776964232s) [noise=true , address=145.79.227.41:4087 , piece_id=GB226GWOX7TC23BAQTUJN6QU2VO7QIONZ5VSJI4NY5ESARCH7WYA, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.908557673 (545h28m40.779640708s) [noise=true , address=145.79.227.9:4365 , piece_id=NAU2HWKLTB7NHLNBGBMHFLTL6S2BH4OPLO6FWWELH2DSPZ6MEZNQ, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.909631207 (545h28m40.780714253s) [noise=true , address=145.79.227.29:4942 , piece_id=ZMNDBC6VLZEBJ2WJ66EX3ML7VHFW7IQLPQTU5EMWIKVDEHIYKVTA, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.910935481 (545h28m40.782018516s) [noise=true , address=145.79.227.45:4229 , piece_id=FZ7D676G2K3DWTMANEAICJU7ECJZWRDLTNCFFKTDFEFGT7ADDX7A, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.913792060 (545h28m40.784875095s) [noise=true , address=145.79.227.17:4971 , piece_id=TAUKFPKSMBYPJM73TXEJJHMYM7ADJW57ZQF4V2ALOU3TJOBVHH4A, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.917421171 (545h28m40.788504216s) [noise=true , address=145.79.227.51:4409 , piece_id=TC6HJ3S7VM6OA6BEXZGM6V43WT7VY7ZP7XTM2ZUIKWAVPCXCPV7Q, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:40.934456098 (545h28m40.805539133s) [noise=true , address=145.79.227.3:4151 , piece_id=Z4AZOB7YRU54ALQIXNHV7K5YXBRQ2VOHPYWAYI2PVK4FZG2VCHAQ, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:41.081958449 (545h28m40.953041494s) [noise=true , address=77.90.31.19:4844 , piece_id=F3LZ7EJOCTGNF4GJBX26AGC4QB4WQ6QHSSXJ6AOQWBXRMZSL34ZA, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:41.131461521 (545h28m41.002544576s) [noise=true , address=45.91.116.9:4261 , piece_id=MMTDQLCP43W5LDA6BGDPVRBQ4E2VQ7GBEPDYOUPXFCDPX3ORRPYA, seg_pos={0 0}, upload=478331]: Done uploading piece. err: <nil> Dec 28 08:57:41.131488533 (545h28m41.002571588s) [seg_pos={0 0}, upload=478331]: Segment reached optimal threshold of 13 pieces. Dec 28 08:57:41.131771992 (545h28m41.002855038s) [noise=true , address=45.91.116.11:4119 , piece_id=5P3HKX2KJRS4GYU2GYWJ5FUAXEWVOB5EIWM3RND6MY3NOMPIBFXA, seg_pos={0 0}, upload=478331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF Dec 28 08:57:41.131782632 (545h28m41.002865677s) [noise=true , address=45.91.116.11:4119 , piece_id=5P3HKX2KJRS4GYU2GYWJ5FUAXEWVOB5EIWM3RND6MY3NOMPIBFXA, seg_pos={0 0}, upload=478331]: Long tail cancellation due to optimal threshold Dec 28 08:57:41.131968947 (545h28m41.003052012s) [noise=true , address=45.91.116.7:4010 , piece_id=PICVBYU3LMFEJ2UTHVQVHMQQB3EYZ25XXAGM6DELW5DLDJ7CHJOQ, seg_pos={0 0}, upload=478331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:44598->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:44598->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44598->45.91.116.7:4010: use of closed network connection Dec 28 08:57:41.132006558 (545h28m41.003089623s) [noise=true , address=45.91.116.7:4010 , piece_id=PICVBYU3LMFEJ2UTHVQVHMQQB3EYZ25XXAGM6DELW5DLDJ7CHJOQ, seg_pos={0 0}, upload=478331]: Long tail cancellation due to optimal threshold Dec 28 08:57:41.132100998 (545h28m41.003184034s) [noise=true , address=45.91.116.5:4297 , piece_id=NQYN7AQNQNF7Z4TJJYUCXKPVVC5I3JXXPPKA3H4V2FJDBL2GBG2A, seg_pos={0 0}, upload=478331]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 28 08:57:41.132113924 (545h28m41.003196959s) [noise=true , address=45.91.116.5:4297 , piece_id=NQYN7AQNQNF7Z4TJJYUCXKPVVC5I3JXXPPKA3H4V2FJDBL2GBG2A, seg_pos={0 0}, upload=478331]: Long tail cancellation due to optimal threshold Dec 28 08:57:41.132140273 (545h28m41.003223318s) [seg_pos={0 0}, upload=478331]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 08:57:41.132230986 (545h28m41.003314031s) [upload=478331]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 08:57:41.132238360 (545h28m41.003321405s) [upload=478331]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 08:57:41.428285145 (545h28m41.29936818s) [upload=478331]: Done uploading segments... Dec 28 08:57:41.428349467 (545h28m41.299432513s) [upload=478331]: Upload finished. err: <nil> Duration: 30.002822345s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [X.XX.XX.......XX.X....XXXX.XX....X....X.XX..XX..X........X.X] duration: 5.00069205s trace: https://jaeger.dev.storj.io/trace/46f1c877a37f9edc FAIL (2025-12-28 08:57:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000781551s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [.X.XX.XX.......XX.X....XXXX.XX....X....X.XX..XX..X........X.] duration: 3.361057599s trace: https://jaeger.dev.storj.io/trace/780b3dc079a52478 FAIL (2025-12-28 08:57:09 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.361083329s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [X.XX.X.XX..X..XX....X...X.XX.XX.......X..X......XXXX.....XX.] duration: 5.879841024s trace: https://jaeger.dev.storj.io/trace/6ff1995b076bde1 FAIL (2025-12-28 08:56:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 08:55:46.483645792 (545h26m46.354728837s) [upload=478304]: Starting upload Dec 28 08:55:46.483722819 (545h26m46.354805854s) [upload=478304]: Done starting upload Dec 28 08:55:46.483756232 (545h26m46.354839267s) [upload=478304]: Uploading segments... Dec 28 08:55:46.483934442 (545h26m46.355017477s) [upload=478304]: Got next segment. Inline: false Dec 28 08:55:46.483966002 (545h26m46.355049037s) [upload=478304]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 08:55:46.483970079 (545h26m46.355053115s) [upload=478304]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 08:55:46.726455434 (545h26m46.597538469s) [seg_pos={0 0}, upload=478304]: Begin upload segment... Dec 28 08:55:46.726496983 (545h26m46.597580018s) [seg_pos={0 0}, upload=478304]: Done begin upload segment. Dec 28 08:55:46.726532360 (545h26m46.597615395s) [noise=true , address=145.79.227.39:4015 , piece_id=PHQPR3ABXFTKKLG6TT3ATSO65Z2Q27S7CBJXDEPHIR2HPIFGGVAA, seg_pos={0 0}, upload=478304]: Uploading piece... Dec 28 08:55:46.727142622 (545h26m46.598225667s) [noise=true , address=45.91.116.6:4974 , piece_id=CD5SPVWPOBHYEMN7AVPJBYCI2DUB7H43ZRONXVQSNM3GHFFNS4PA, seg_pos={0 0}, upload=478304]: Uploading piece... Dec 28 08:55:46.727958846 (545h26m46.599041881s) [noise=true , address=145.79.227.12:4767 , piece_id=7RXIBDD3SIYNCSNMOIILBUC45JRYC2YFTSAXDLMZWN6GI3CUOJ2A, seg_pos={0 0}, upload=478304]: Uploading piece... Dec 28 08:55:46.728356845 (545h26m46.59943989s) [noise=true , address=145.79.227.2:4115 , piece_id=QX5JTS2SNNXDPFIGOER7RDOAS3527UXJJ3VYELVGE5VOSN7IRBCA, seg_pos={0 0}, upload=478304]: Uploading piece... Dec 28 08:55:46.728770833 (545h26m46.599853878s) [noise=true , address=145.79.227.19:4079 , piece_id=2L4VY5F7RPWPXNJ7BQB76D2O4CYEZKDGDPC5OOGVWC6W232AJJVQ, seg_pos={0 0}, upload=478304]: Uploading piece... Dec 28 08:55:46.729116241 (545h26m46.600199286s) [noise=true , address=145.79.227.9:4365 , piece_id=ZJWVG67QWCP64SDK2QQFK6DMO7E7ACVODWSSVDCPWK2I54WEYZVQ, seg_pos={0 0}, upload=478304]: Uploading piece... Dec 28 08:55:46.729411603 (545h26m46.600494638s) [noise=true , address=45.91.116.8:4046 , piece_id=RXI2XWOHQ3KNOOLUIMO5MVXBIY5RD7U6YQAWTZPYRV6S66MARK7Q, seg_pos={0 0}, upload=478304]: Uploading piece... Dec 28 08:55:46.730636084 (545h26m46.601719119s) [noise=true , address=77.90.31.15:4700 , piece_id=NOK2YSRJAVR3IFIUNIZGRMMVB76SKLIWBQLJKMRTVETWTLNTEOVA, seg_pos={0 0}, upload=478304]: Uploading piece... Dec 28 08:55:46.731799330 (545h26m46.602882375s) [noise=true , address=145.79.227.47:4266 , piece_id=MMH54XWUOPXLMCFXLXDVUJ53RPYE2UR55JFOQ2RQC5LPBRJNKEOQ, seg_pos={0 0}, upload=478304]: Uploading piece... Dec 28 08:55:46.732866541 (545h26m46.603949576s) [noise=true , address=77.90.31.11:4556 , piece_id=3O7VPJP5KU6SM2NST5DI5WC4KUQXOMOOPGXMPQH5WI4KVOQ4A4NQ, seg_pos={0 0}, upload=478304]: Uploading piece... Dec 28 08:55:46.734146599 (545h26m46.605229634s) [noise=true , address=145.79.227.20:4437 , piece_id=DVEUB67MAPBGH3YIUBEV7IOBXK4LJWZS7KBNRQRIIUPXR32IAD5Q, seg_pos={0 0}, upload=478304]: Uploading piece... Dec 28 08:55:46.734886427 (545h26m46.605969463s) [noise=true , address=45.91.116.4:4939 , piece_id=BWN4KCDWGG3ERUSU5ZW6657UXYTLZ5JS3O3NJ35DVP2FYUDZ6CWA, seg_pos={0 0}, upload=478304]: Uploading piece... Dec 28 08:55:46.736304338 (545h26m46.607387373s) [noise=true , address=77.90.31.16:4736 , piece_id=C3YZYFBJVP5GPGKZEQWIMLABMCDSUGBU5YXT622XWXJBQKG5UMSA, seg_pos={0 0}, upload=478304]: Uploading piece... Dec 28 08:55:46.736480413 (545h26m46.607563448s) [noise=true , address=145.79.227.5:4223 , piece_id=OTXYX3D7QJRIVXP4BFBWGU5CSZTNY7MZ5EZ6OKQURNZRWCA2KKVQ, seg_pos={0 0}, upload=478304]: Uploading piece... Dec 28 08:55:46.737620825 (545h26m46.60870386s) [noise=true , address=145.79.227.50:4375 , piece_id=UDADL6QDIOYIYIAJ4IZMC4BN3CCIZNKXLZPR5ZLNZDANWUKM3RVQ, seg_pos={0 0}, upload=478304]: Uploading piece... Dec 28 08:55:46.738972178 (545h26m46.610055213s) [noise=true , address=45.91.116.14:4192 , piece_id=7ELA7DUZA4WZQZ6FUESXQH4RN6AOBBMHHXFNFTJSXPZUCGSP2GOQ, seg_pos={0 0}, upload=478304]: Uploading piece... Dec 28 08:55:48.448356083 (545h26m48.319439118s) [upload=478304]: Next returned nil segment Dec 28 08:55:48.448391300 (545h26m48.319474335s) [upload=478304]: Waiting for error group managing segments... Dec 28 08:55:48.724923425 (545h26m48.59600646s) [noise=true , address=145.79.227.12:4767 , piece_id=7RXIBDD3SIYNCSNMOIILBUC45JRYC2YFTSAXDLMZWN6GI3CUOJ2A, seg_pos={0 0}, upload=478304]: Done uploading piece. err: <nil> Dec 28 08:55:48.738664851 (545h26m48.609747897s) [noise=true , address=145.79.227.2:4115 , piece_id=QX5JTS2SNNXDPFIGOER7RDOAS3527UXJJ3VYELVGE5VOSN7IRBCA, seg_pos={0 0}, upload=478304]: Done uploading piece. err: <nil> Dec 28 08:55:48.741484951 (545h26m48.612567996s) [noise=true , address=145.79.227.9:4365 , piece_id=ZJWVG67QWCP64SDK2QQFK6DMO7E7ACVODWSSVDCPWK2I54WEYZVQ, seg_pos={0 0}, upload=478304]: Done uploading piece. err: <nil> Dec 28 08:55:48.748723445 (545h26m48.61980648s) [noise=true , address=145.79.227.39:4015 , piece_id=PHQPR3ABXFTKKLG6TT3ATSO65Z2Q27S7CBJXDEPHIR2HPIFGGVAA, seg_pos={0 0}, upload=478304]: Done uploading piece. err: <nil> Dec 28 08:55:48.756495594 (545h26m48.627578639s) [noise=true , address=145.79.227.47:4266 , piece_id=MMH54XWUOPXLMCFXLXDVUJ53RPYE2UR55JFOQ2RQC5LPBRJNKEOQ, seg_pos={0 0}, upload=478304]: Done uploading piece. err: <nil> Dec 28 08:55:48.756526493 (545h26m48.627609528s) [noise=true , address=145.79.227.19:4079 , piece_id=2L4VY5F7RPWPXNJ7BQB76D2O4CYEZKDGDPC5OOGVWC6W232AJJVQ, seg_pos={0 0}, upload=478304]: Done uploading piece. err: <nil> Dec 28 08:55:48.757229232 (545h26m48.628312267s) [noise=true , address=145.79.227.50:4375 , piece_id=UDADL6QDIOYIYIAJ4IZMC4BN3CCIZNKXLZPR5ZLNZDANWUKM3RVQ, seg_pos={0 0}, upload=478304]: Done uploading piece. err: <nil> Dec 28 08:55:48.760761177 (545h26m48.631844222s) [noise=true , address=145.79.227.20:4437 , piece_id=DVEUB67MAPBGH3YIUBEV7IOBXK4LJWZS7KBNRQRIIUPXR32IAD5Q, seg_pos={0 0}, upload=478304]: Done uploading piece. err: <nil> Dec 28 08:55:48.794035166 (545h26m48.665118202s) [noise=true , address=145.79.227.5:4223 , piece_id=OTXYX3D7QJRIVXP4BFBWGU5CSZTNY7MZ5EZ6OKQURNZRWCA2KKVQ, seg_pos={0 0}, upload=478304]: Done uploading piece. err: <nil> Dec 28 08:55:49.004400188 (545h26m48.875483253s) [noise=true , address=45.91.116.6:4974 , piece_id=CD5SPVWPOBHYEMN7AVPJBYCI2DUB7H43ZRONXVQSNM3GHFFNS4PA, seg_pos={0 0}, upload=478304]: Done uploading piece. err: <nil> Dec 28 08:55:49.017261929 (545h26m48.888344964s) [noise=true , address=77.90.31.11:4556 , piece_id=3O7VPJP5KU6SM2NST5DI5WC4KUQXOMOOPGXMPQH5WI4KVOQ4A4NQ, seg_pos={0 0}, upload=478304]: Done uploading piece. err: <nil> Dec 28 08:55:49.025850744 (545h26m48.896933779s) [noise=true , address=77.90.31.16:4736 , piece_id=C3YZYFBJVP5GPGKZEQWIMLABMCDSUGBU5YXT622XWXJBQKG5UMSA, seg_pos={0 0}, upload=478304]: Done uploading piece. err: <nil> Dec 28 08:55:49.106259496 (545h26m48.977342531s) [noise=true , address=45.91.116.4:4939 , piece_id=BWN4KCDWGG3ERUSU5ZW6657UXYTLZ5JS3O3NJ35DVP2FYUDZ6CWA, seg_pos={0 0}, upload=478304]: Done uploading piece. err: <nil> Dec 28 08:55:49.106280175 (545h26m48.97736322s) [seg_pos={0 0}, upload=478304]: Segment reached optimal threshold of 13 pieces. Dec 28 08:55:49.106487089 (545h26m48.977570124s) [noise=true , address=45.91.116.14:4192 , piece_id=7ELA7DUZA4WZQZ6FUESXQH4RN6AOBBMHHXFNFTJSXPZUCGSP2GOQ, seg_pos={0 0}, upload=478304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 28 08:55:49.106493341 (545h26m48.977576376s) [noise=true , address=45.91.116.14:4192 , piece_id=7ELA7DUZA4WZQZ6FUESXQH4RN6AOBBMHHXFNFTJSXPZUCGSP2GOQ, seg_pos={0 0}, upload=478304]: Long tail cancellation due to optimal threshold Dec 28 08:55:49.106549438 (545h26m48.977632473s) [noise=true , address=77.90.31.15:4700 , piece_id=NOK2YSRJAVR3IFIUNIZGRMMVB76SKLIWBQLJKMRTVETWTLNTEOVA, seg_pos={0 0}, upload=478304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 28 08:55:49.106563494 (545h26m48.97764654s) [noise=true , address=77.90.31.15:4700 , piece_id=NOK2YSRJAVR3IFIUNIZGRMMVB76SKLIWBQLJKMRTVETWTLNTEOVA, seg_pos={0 0}, upload=478304]: Long tail cancellation due to optimal threshold Dec 28 08:55:49.106789375 (545h26m48.97787242s) [noise=true , address=45.91.116.8:4046 , piece_id=RXI2XWOHQ3KNOOLUIMO5MVXBIY5RD7U6YQAWTZPYRV6S66MARK7Q, seg_pos={0 0}, upload=478304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 28 08:55:49.106854088 (545h26m48.977937164s) [noise=true , address=45.91.116.8:4046 , piece_id=RXI2XWOHQ3KNOOLUIMO5MVXBIY5RD7U6YQAWTZPYRV6S66MARK7Q, seg_pos={0 0}, upload=478304]: Long tail cancellation due to optimal threshold Dec 28 08:55:49.106896218 (545h26m48.977979253s) [seg_pos={0 0}, upload=478304]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 08:55:49.107022579 (545h26m48.978105614s) [upload=478304]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 08:55:49.107028120 (545h26m48.978111165s) [upload=478304]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 08:55:49.409681595 (545h26m49.28076463s) [upload=478304]: Done uploading segments... Dec 28 08:55:49.409700871 (545h26m49.280783906s) [upload=478304]: Upload finished. err: <nil> Duration: 5.879875268s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [..X.XX.XX.......XX.X....XXXX.XX....X....X.XX..XX..X........X] duration: 5.000980468s trace: https://jaeger.dev.storj.io/trace/780b3dc079a52478 FAIL (2025-12-28 08:56:53 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001000897s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [...X.XX.XX.......XX.X....XXXX.XX....X....X.XX..XX..X........] duration: 3.446766963s trace: https://jaeger.dev.storj.io/trace/77e06878e3b86651 FAIL (2025-12-28 08:56:31 UTC) orbiter warning: context deadline exceeded Duration: 3.446786401s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.X.XX.X.XX..X..XX....X...X.XX.XX.......X..X......XXXX.....XX] duration: 30.002260985s trace: https://jaeger.dev.storj.io/trace/6ff1995b076bde1 FAIL (2025-12-28 08:56:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 08:55:46.483645792 (545h26m46.354728837s) [upload=478304]: Starting upload Dec 28 08:55:46.483722819 (545h26m46.354805854s) [upload=478304]: Done starting upload Dec 28 08:55:46.483756232 (545h26m46.354839267s) [upload=478304]: Uploading segments... Dec 28 08:55:46.483934442 (545h26m46.355017477s) [upload=478304]: Got next segment. Inline: false Dec 28 08:55:46.483966002 (545h26m46.355049037s) [upload=478304]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 08:55:46.483970079 (545h26m46.355053115s) [upload=478304]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 08:55:46.726455434 (545h26m46.597538469s) [seg_pos={0 0}, upload=478304]: Begin upload segment... Dec 28 08:55:46.726496983 (545h26m46.597580018s) [seg_pos={0 0}, upload=478304]: Done begin upload segment. Dec 28 08:55:46.726532360 (545h26m46.597615395s) [noise=true , address=145.79.227.39:4015 , piece_id=PHQPR3ABXFTKKLG6TT3ATSO65Z2Q27S7CBJXDEPHIR2HPIFGGVAA, seg_pos={0 0}, upload=478304]: Uploading piece... Dec 28 08:55:46.727142622 (545h26m46.598225667s) [noise=true , address=45.91.116.6:4974 , piece_id=CD5SPVWPOBHYEMN7AVPJBYCI2DUB7H43ZRONXVQSNM3GHFFNS4PA, seg_pos={0 0}, upload=478304]: Uploading piece... Dec 28 08:55:46.727958846 (545h26m46.599041881s) [noise=true , address=145.79.227.12:4767 , piece_id=7RXIBDD3SIYNCSNMOIILBUC45JRYC2YFTSAXDLMZWN6GI3CUOJ2A, seg_pos={0 0}, upload=478304]: Uploading piece... Dec 28 08:55:46.728356845 (545h26m46.59943989s) [noise=true , address=145.79.227.2:4115 , piece_id=QX5JTS2SNNXDPFIGOER7RDOAS3527UXJJ3VYELVGE5VOSN7IRBCA, seg_pos={0 0}, upload=478304]: Uploading piece... Dec 28 08:55:46.728770833 (545h26m46.599853878s) [noise=true , address=145.79.227.19:4079 , piece_id=2L4VY5F7RPWPXNJ7BQB76D2O4CYEZKDGDPC5OOGVWC6W232AJJVQ, seg_pos={0 0}, upload=478304]: Uploading piece... Dec 28 08:55:46.729116241 (545h26m46.600199286s) [noise=true , address=145.79.227.9:4365 , piece_id=ZJWVG67QWCP64SDK2QQFK6DMO7E7ACVODWSSVDCPWK2I54WEYZVQ, seg_pos={0 0}, upload=478304]: Uploading piece... Dec 28 08:55:46.729411603 (545h26m46.600494638s) [noise=true , address=45.91.116.8:4046 , piece_id=RXI2XWOHQ3KNOOLUIMO5MVXBIY5RD7U6YQAWTZPYRV6S66MARK7Q, seg_pos={0 0}, upload=478304]: Uploading piece... Dec 28 08:55:46.730636084 (545h26m46.601719119s) [noise=true , address=77.90.31.15:4700 , piece_id=NOK2YSRJAVR3IFIUNIZGRMMVB76SKLIWBQLJKMRTVETWTLNTEOVA, seg_pos={0 0}, upload=478304]: Uploading piece... Dec 28 08:55:46.731799330 (545h26m46.602882375s) [noise=true , address=145.79.227.47:4266 , piece_id=MMH54XWUOPXLMCFXLXDVUJ53RPYE2UR55JFOQ2RQC5LPBRJNKEOQ, seg_pos={0 0}, upload=478304]: Uploading piece... Dec 28 08:55:46.732866541 (545h26m46.603949576s) [noise=true , address=77.90.31.11:4556 , piece_id=3O7VPJP5KU6SM2NST5DI5WC4KUQXOMOOPGXMPQH5WI4KVOQ4A4NQ, seg_pos={0 0}, upload=478304]: Uploading piece... Dec 28 08:55:46.734146599 (545h26m46.605229634s) [noise=true , address=145.79.227.20:4437 , piece_id=DVEUB67MAPBGH3YIUBEV7IOBXK4LJWZS7KBNRQRIIUPXR32IAD5Q, seg_pos={0 0}, upload=478304]: Uploading piece... Dec 28 08:55:46.734886427 (545h26m46.605969463s) [noise=true , address=45.91.116.4:4939 , piece_id=BWN4KCDWGG3ERUSU5ZW6657UXYTLZ5JS3O3NJ35DVP2FYUDZ6CWA, seg_pos={0 0}, upload=478304]: Uploading piece... Dec 28 08:55:46.736304338 (545h26m46.607387373s) [noise=true , address=77.90.31.16:4736 , piece_id=C3YZYFBJVP5GPGKZEQWIMLABMCDSUGBU5YXT622XWXJBQKG5UMSA, seg_pos={0 0}, upload=478304]: Uploading piece... Dec 28 08:55:46.736480413 (545h26m46.607563448s) [noise=true , address=145.79.227.5:4223 , piece_id=OTXYX3D7QJRIVXP4BFBWGU5CSZTNY7MZ5EZ6OKQURNZRWCA2KKVQ, seg_pos={0 0}, upload=478304]: Uploading piece... Dec 28 08:55:46.737620825 (545h26m46.60870386s) [noise=true , address=145.79.227.50:4375 , piece_id=UDADL6QDIOYIYIAJ4IZMC4BN3CCIZNKXLZPR5ZLNZDANWUKM3RVQ, seg_pos={0 0}, upload=478304]: Uploading piece... Dec 28 08:55:46.738972178 (545h26m46.610055213s) [noise=true , address=45.91.116.14:4192 , piece_id=7ELA7DUZA4WZQZ6FUESXQH4RN6AOBBMHHXFNFTJSXPZUCGSP2GOQ, seg_pos={0 0}, upload=478304]: Uploading piece... Dec 28 08:55:48.448356083 (545h26m48.319439118s) [upload=478304]: Next returned nil segment Dec 28 08:55:48.448391300 (545h26m48.319474335s) [upload=478304]: Waiting for error group managing segments... Dec 28 08:55:48.724923425 (545h26m48.59600646s) [noise=true , address=145.79.227.12:4767 , piece_id=7RXIBDD3SIYNCSNMOIILBUC45JRYC2YFTSAXDLMZWN6GI3CUOJ2A, seg_pos={0 0}, upload=478304]: Done uploading piece. err: <nil> Dec 28 08:55:48.738664851 (545h26m48.609747897s) [noise=true , address=145.79.227.2:4115 , piece_id=QX5JTS2SNNXDPFIGOER7RDOAS3527UXJJ3VYELVGE5VOSN7IRBCA, seg_pos={0 0}, upload=478304]: Done uploading piece. err: <nil> Dec 28 08:55:48.741484951 (545h26m48.612567996s) [noise=true , address=145.79.227.9:4365 , piece_id=ZJWVG67QWCP64SDK2QQFK6DMO7E7ACVODWSSVDCPWK2I54WEYZVQ, seg_pos={0 0}, upload=478304]: Done uploading piece. err: <nil> Dec 28 08:55:48.748723445 (545h26m48.61980648s) [noise=true , address=145.79.227.39:4015 , piece_id=PHQPR3ABXFTKKLG6TT3ATSO65Z2Q27S7CBJXDEPHIR2HPIFGGVAA, seg_pos={0 0}, upload=478304]: Done uploading piece. err: <nil> Dec 28 08:55:48.756495594 (545h26m48.627578639s) [noise=true , address=145.79.227.47:4266 , piece_id=MMH54XWUOPXLMCFXLXDVUJ53RPYE2UR55JFOQ2RQC5LPBRJNKEOQ, seg_pos={0 0}, upload=478304]: Done uploading piece. err: <nil> Dec 28 08:55:48.756526493 (545h26m48.627609528s) [noise=true , address=145.79.227.19:4079 , piece_id=2L4VY5F7RPWPXNJ7BQB76D2O4CYEZKDGDPC5OOGVWC6W232AJJVQ, seg_pos={0 0}, upload=478304]: Done uploading piece. err: <nil> Dec 28 08:55:48.757229232 (545h26m48.628312267s) [noise=true , address=145.79.227.50:4375 , piece_id=UDADL6QDIOYIYIAJ4IZMC4BN3CCIZNKXLZPR5ZLNZDANWUKM3RVQ, seg_pos={0 0}, upload=478304]: Done uploading piece. err: <nil> Dec 28 08:55:48.760761177 (545h26m48.631844222s) [noise=true , address=145.79.227.20:4437 , piece_id=DVEUB67MAPBGH3YIUBEV7IOBXK4LJWZS7KBNRQRIIUPXR32IAD5Q, seg_pos={0 0}, upload=478304]: Done uploading piece. err: <nil> Dec 28 08:55:48.794035166 (545h26m48.665118202s) [noise=true , address=145.79.227.5:4223 , piece_id=OTXYX3D7QJRIVXP4BFBWGU5CSZTNY7MZ5EZ6OKQURNZRWCA2KKVQ, seg_pos={0 0}, upload=478304]: Done uploading piece. err: <nil> Dec 28 08:55:49.004400188 (545h26m48.875483253s) [noise=true , address=45.91.116.6:4974 , piece_id=CD5SPVWPOBHYEMN7AVPJBYCI2DUB7H43ZRONXVQSNM3GHFFNS4PA, seg_pos={0 0}, upload=478304]: Done uploading piece. err: <nil> Dec 28 08:55:49.017261929 (545h26m48.888344964s) [noise=true , address=77.90.31.11:4556 , piece_id=3O7VPJP5KU6SM2NST5DI5WC4KUQXOMOOPGXMPQH5WI4KVOQ4A4NQ, seg_pos={0 0}, upload=478304]: Done uploading piece. err: <nil> Dec 28 08:55:49.025850744 (545h26m48.896933779s) [noise=true , address=77.90.31.16:4736 , piece_id=C3YZYFBJVP5GPGKZEQWIMLABMCDSUGBU5YXT622XWXJBQKG5UMSA, seg_pos={0 0}, upload=478304]: Done uploading piece. err: <nil> Dec 28 08:55:49.106259496 (545h26m48.977342531s) [noise=true , address=45.91.116.4:4939 , piece_id=BWN4KCDWGG3ERUSU5ZW6657UXYTLZ5JS3O3NJ35DVP2FYUDZ6CWA, seg_pos={0 0}, upload=478304]: Done uploading piece. err: <nil> Dec 28 08:55:49.106280175 (545h26m48.97736322s) [seg_pos={0 0}, upload=478304]: Segment reached optimal threshold of 13 pieces. Dec 28 08:55:49.106487089 (545h26m48.977570124s) [noise=true , address=45.91.116.14:4192 , piece_id=7ELA7DUZA4WZQZ6FUESXQH4RN6AOBBMHHXFNFTJSXPZUCGSP2GOQ, seg_pos={0 0}, upload=478304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: expected piece hash; context canceled; EOF Dec 28 08:55:49.106493341 (545h26m48.977576376s) [noise=true , address=45.91.116.14:4192 , piece_id=7ELA7DUZA4WZQZ6FUESXQH4RN6AOBBMHHXFNFTJSXPZUCGSP2GOQ, seg_pos={0 0}, upload=478304]: Long tail cancellation due to optimal threshold Dec 28 08:55:49.106549438 (545h26m48.977632473s) [noise=true , address=77.90.31.15:4700 , piece_id=NOK2YSRJAVR3IFIUNIZGRMMVB76SKLIWBQLJKMRTVETWTLNTEOVA, seg_pos={0 0}, upload=478304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 28 08:55:49.106563494 (545h26m48.97764654s) [noise=true , address=77.90.31.15:4700 , piece_id=NOK2YSRJAVR3IFIUNIZGRMMVB76SKLIWBQLJKMRTVETWTLNTEOVA, seg_pos={0 0}, upload=478304]: Long tail cancellation due to optimal threshold Dec 28 08:55:49.106789375 (545h26m48.97787242s) [noise=true , address=45.91.116.8:4046 , piece_id=RXI2XWOHQ3KNOOLUIMO5MVXBIY5RD7U6YQAWTZPYRV6S66MARK7Q, seg_pos={0 0}, upload=478304]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: expected piece hash; context canceled; EOF Dec 28 08:55:49.106854088 (545h26m48.977937164s) [noise=true , address=45.91.116.8:4046 , piece_id=RXI2XWOHQ3KNOOLUIMO5MVXBIY5RD7U6YQAWTZPYRV6S66MARK7Q, seg_pos={0 0}, upload=478304]: Long tail cancellation due to optimal threshold Dec 28 08:55:49.106896218 (545h26m48.977979253s) [seg_pos={0 0}, upload=478304]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 08:55:49.107022579 (545h26m48.978105614s) [upload=478304]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 08:55:49.107028120 (545h26m48.978111165s) [upload=478304]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 08:55:49.409681595 (545h26m49.28076463s) [upload=478304]: Done uploading segments... Dec 28 08:55:49.409700871 (545h26m49.280783906s) [upload=478304]: Upload finished. err: <nil> Duration: 30.00227927s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [..X.XX.X.XX..X..XX....X...X.XX.XX.......X..X......XXXX.....X] duration: 30.003216336s trace: https://jaeger.dev.storj.io/trace/5d977ed2cf1e580 FAIL (2025-12-28 08:55:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 08:55:12.298197618 (545h26m12.169280653s) [upload=478298]: Starting upload Dec 28 08:55:12.298256811 (545h26m12.169339846s) [upload=478298]: Done starting upload Dec 28 08:55:12.298345960 (545h26m12.169428995s) [upload=478298]: Uploading segments... Dec 28 08:55:12.298445630 (545h26m12.169528675s) [upload=478298]: Got next segment. Inline: false Dec 28 08:55:12.298489233 (545h26m12.169572268s) [upload=478298]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 08:55:12.298494182 (545h26m12.169577217s) [upload=478298]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 08:55:12.540527466 (545h26m12.411610511s) [seg_pos={0 0}, upload=478298]: Begin upload segment... Dec 28 08:55:12.540588592 (545h26m12.411671627s) [seg_pos={0 0}, upload=478298]: Done begin upload segment. Dec 28 08:55:12.540607168 (545h26m12.411690213s) [noise=true , address=77.90.31.10:4518 , piece_id=6D2GSCZJ6WKYNSRZJ7HN2QVDAGVFSUEAOXG3TZCUC7VV2AOILWTQ, seg_pos={0 0}, upload=478298]: Uploading piece... Dec 28 08:55:12.540623178 (545h26m12.411706213s) [noise=true , address=145.79.227.2:4115 , piece_id=YOADLIRH7ZKAY47HFKF6CZDYNYAIEL3YMNF3YNQ7E52QG76PRIPA, seg_pos={0 0}, upload=478298]: Uploading piece... Dec 28 08:55:12.541152276 (545h26m12.412235321s) [noise=true , address=45.91.116.11:4119 , piece_id=O3PD5LRRWWHXBZ4GNEKLGPFPGGVEBTRRJG7LQFUUEBHTKCBZQDJQ, seg_pos={0 0}, upload=478298]: Uploading piece... Dec 28 08:55:12.541554792 (545h26m12.412637837s) [noise=true , address=77.90.31.13:4629 , piece_id=WMJUHZUYXAWAYQ4TTQLC4F3UU5XKEIVQOYK3NRTZZRRB4CZHMSOA, seg_pos={0 0}, upload=478298]: Uploading piece... Dec 28 08:55:12.542157059 (545h26m12.413240104s) [noise=true , address=45.91.116.13:4156 , piece_id=GL3J7TLZW7Z43BOQTPJJYBWODWG3IBKFRZDQQVQBJ3IAMNNL5VNQ, seg_pos={0 0}, upload=478298]: Uploading piece... Dec 28 08:55:12.542571849 (545h26m12.413654884s) [noise=true , address=145.79.227.11:4729 , piece_id=5GSOTY64DUT4JTPJREVNI3C7USBRRPJMHCPOVRITXATLFNTZTVOA, seg_pos={0 0}, upload=478298]: Uploading piece... Dec 28 08:55:12.543054848 (545h26m12.414137883s) [noise=true , address=77.90.31.11:4555 , piece_id=ADSYEAHRB2WLUABSLFW2EG5DJFKHYSX6UZBAEFUDXAQTB2QMVGCQ, seg_pos={0 0}, upload=478298]: Uploading piece... Dec 28 08:55:12.543498734 (545h26m12.414581779s) [noise=true , address=145.79.227.31:4797 , piece_id=2OMCB4HMNEDSXZERUYONUDHNTXTZGDLVSEKPUI4LTJ5OJYI4DMIA, seg_pos={0 0}, upload=478298]: Uploading piece... Dec 28 08:55:12.545174144 (545h26m12.416257179s) [noise=true , address=145.79.227.45:4230 , piece_id=MDGUFPXE5BNEUYXACHGKDN3OZ2LKPAO6DYAPQYMQRJLZWVJOPJ6A, seg_pos={0 0}, upload=478298]: Uploading piece... Dec 28 08:55:12.546928956 (545h26m12.418012001s) [noise=true , address=45.91.116.10:4084 , piece_id=53WLLAQM55MUWCMAVY4IUK2HODPVK3FOCWS36KWHNALRSTTBL6UA, seg_pos={0 0}, upload=478298]: Uploading piece... Dec 28 08:55:12.548479197 (545h26m12.419562242s) [noise=true , address=77.90.31.21:4880 , piece_id=BJLBMSXZH3M6PQ3CRYLI6TZVCNEWTLQICCC2AXCW62OILIGXNUDQ, seg_pos={0 0}, upload=478298]: Uploading piece... Dec 28 08:55:12.549187235 (545h26m12.42027027s) [noise=true , address=145.79.227.52:4446 , piece_id=5ULWKXVZISMKZZ5KUBDU46PNJ3V53DAL7SEAFOEHRO4W3FUR3K3Q, seg_pos={0 0}, upload=478298]: Uploading piece... Dec 28 08:55:12.551017311 (545h26m12.422100346s) [noise=true , address=145.79.227.47:4265 , piece_id=DPI3V4AM57FBXERX6KY5BQU3G65QHL2NNQPZR23ZZGSM6PRSYAXA, seg_pos={0 0}, upload=478298]: Uploading piece... Dec 28 08:55:12.551823015 (545h26m12.42290605s) [noise=true , address=77.90.31.15:4700 , piece_id=FGJUAE6X4GOWJ4JY7QJIPRS5PYAYDLKTV2TBBYZFJ77JU5PBG3WQ, seg_pos={0 0}, upload=478298]: Uploading piece... Dec 28 08:55:12.554199690 (545h26m12.425282735s) [noise=true , address=145.79.227.51:4409 , piece_id=UHHUDVIEGNW5XKY6OAS3SZJAJIC6E2OEWR65VFOG6LWPV3GXUYQA, seg_pos={0 0}, upload=478298]: Uploading piece... Dec 28 08:55:12.555729844 (545h26m12.426812889s) [noise=true , address=145.79.227.32:4836 , piece_id=J7PNNLZSHO5U5UDT7MQHL67BBUUA7OUD34WWSTV3J5PPZABFWPOA, seg_pos={0 0}, upload=478298]: Uploading piece... Dec 28 08:55:14.272556311 (545h26m14.143639346s) [upload=478298]: Next returned nil segment Dec 28 08:55:14.272572040 (545h26m14.143655075s) [upload=478298]: Waiting for error group managing segments... Dec 28 08:55:14.549069741 (545h26m14.420152776s) [noise=true , address=145.79.227.2:4115 , piece_id=YOADLIRH7ZKAY47HFKF6CZDYNYAIEL3YMNF3YNQ7E52QG76PRIPA, seg_pos={0 0}, upload=478298]: Done uploading piece. err: <nil> Dec 28 08:55:14.563521279 (545h26m14.434604324s) [noise=true , address=145.79.227.52:4446 , piece_id=5ULWKXVZISMKZZ5KUBDU46PNJ3V53DAL7SEAFOEHRO4W3FUR3K3Q, seg_pos={0 0}, upload=478298]: Done uploading piece. err: <nil> Dec 28 08:55:14.563641217 (545h26m14.434724262s) [noise=true , address=145.79.227.11:4729 , piece_id=5GSOTY64DUT4JTPJREVNI3C7USBRRPJMHCPOVRITXATLFNTZTVOA, seg_pos={0 0}, upload=478298]: Done uploading piece. err: <nil> Dec 28 08:55:14.564207085 (545h26m14.43529013s) [noise=true , address=145.79.227.51:4409 , piece_id=UHHUDVIEGNW5XKY6OAS3SZJAJIC6E2OEWR65VFOG6LWPV3GXUYQA, seg_pos={0 0}, upload=478298]: Done uploading piece. err: <nil> Dec 28 08:55:14.573576356 (545h26m14.444659401s) [noise=true , address=145.79.227.47:4265 , piece_id=DPI3V4AM57FBXERX6KY5BQU3G65QHL2NNQPZR23ZZGSM6PRSYAXA, seg_pos={0 0}, upload=478298]: Done uploading piece. err: <nil> Dec 28 08:55:14.579964469 (545h26m14.451047514s) [noise=true , address=145.79.227.32:4836 , piece_id=J7PNNLZSHO5U5UDT7MQHL67BBUUA7OUD34WWSTV3J5PPZABFWPOA, seg_pos={0 0}, upload=478298]: Done uploading piece. err: <nil> Dec 28 08:55:14.581642415 (545h26m14.45272546s) [noise=true , address=145.79.227.31:4797 , piece_id=2OMCB4HMNEDSXZERUYONUDHNTXTZGDLVSEKPUI4LTJ5OJYI4DMIA, seg_pos={0 0}, upload=478298]: Done uploading piece. err: <nil> Dec 28 08:55:14.589997705 (545h26m14.46108075s) [noise=true , address=145.79.227.45:4230 , piece_id=MDGUFPXE5BNEUYXACHGKDN3OZ2LKPAO6DYAPQYMQRJLZWVJOPJ6A, seg_pos={0 0}, upload=478298]: Done uploading piece. err: <nil> Dec 28 08:55:14.709324444 (545h26m14.580407479s) [noise=true , address=77.90.31.10:4518 , piece_id=6D2GSCZJ6WKYNSRZJ7HN2QVDAGVFSUEAOXG3TZCUC7VV2AOILWTQ, seg_pos={0 0}, upload=478298]: Done uploading piece. err: <nil> Dec 28 08:55:14.819084145 (545h26m14.69016718s) [noise=true , address=77.90.31.13:4629 , piece_id=WMJUHZUYXAWAYQ4TTQLC4F3UU5XKEIVQOYK3NRTZZRRB4CZHMSOA, seg_pos={0 0}, upload=478298]: Done uploading piece. err: <nil> Dec 28 08:55:14.863716763 (545h26m14.734799798s) [noise=true , address=77.90.31.11:4555 , piece_id=ADSYEAHRB2WLUABSLFW2EG5DJFKHYSX6UZBAEFUDXAQTB2QMVGCQ, seg_pos={0 0}, upload=478298]: Done uploading piece. err: <nil> Dec 28 08:55:14.877680663 (545h26m14.748763698s) [noise=true , address=77.90.31.15:4700 , piece_id=FGJUAE6X4GOWJ4JY7QJIPRS5PYAYDLKTV2TBBYZFJ77JU5PBG3WQ, seg_pos={0 0}, upload=478298]: Done uploading piece. err: <nil> Dec 28 08:55:14.934567346 (545h26m14.805650381s) [noise=true , address=45.91.116.11:4119 , piece_id=O3PD5LRRWWHXBZ4GNEKLGPFPGGVEBTRRJG7LQFUUEBHTKCBZQDJQ, seg_pos={0 0}, upload=478298]: Done uploading piece. err: <nil> Dec 28 08:55:14.934592032 (545h26m14.805675077s) [seg_pos={0 0}, upload=478298]: Segment reached optimal threshold of 13 pieces. Dec 28 08:55:14.934858760 (545h26m14.805941805s) [noise=true , address=77.90.31.21:4880 , piece_id=BJLBMSXZH3M6PQ3CRYLI6TZVCNEWTLQICCC2AXCW62OILIGXNUDQ, seg_pos={0 0}, upload=478298]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: expected piece hash; context canceled; EOF Dec 28 08:55:14.934867767 (545h26m14.805950802s) [noise=true , address=77.90.31.21:4880 , piece_id=BJLBMSXZH3M6PQ3CRYLI6TZVCNEWTLQICCC2AXCW62OILIGXNUDQ, seg_pos={0 0}, upload=478298]: Long tail cancellation due to optimal threshold Dec 28 08:55:14.934986524 (545h26m14.806069559s) [noise=true , address=45.91.116.13:4156 , piece_id=GL3J7TLZW7Z43BOQTPJJYBWODWG3IBKFRZDQQVQBJ3IAMNNL5VNQ, seg_pos={0 0}, upload=478298]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:51102->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:51102->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51102->45.91.116.13:4156: use of closed network connection Dec 28 08:55:14.935012402 (545h26m14.806095437s) [noise=true , address=45.91.116.13:4156 , piece_id=GL3J7TLZW7Z43BOQTPJJYBWODWG3IBKFRZDQQVQBJ3IAMNNL5VNQ, seg_pos={0 0}, upload=478298]: Long tail cancellation due to optimal threshold Dec 28 08:55:14.935108816 (545h26m14.806191971s) [noise=true , address=45.91.116.10:4084 , piece_id=53WLLAQM55MUWCMAVY4IUK2HODPVK3FOCWS36KWHNALRSTTBL6UA, seg_pos={0 0}, upload=478298]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF Dec 28 08:55:14.935146448 (545h26m14.806229493s) [noise=true , address=45.91.116.10:4084 , piece_id=53WLLAQM55MUWCMAVY4IUK2HODPVK3FOCWS36KWHNALRSTTBL6UA, seg_pos={0 0}, upload=478298]: Long tail cancellation due to optimal threshold Dec 28 08:55:14.935186333 (545h26m14.806269369s) [seg_pos={0 0}, upload=478298]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 08:55:14.935317844 (545h26m14.806400889s) [upload=478298]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 08:55:14.935324927 (545h26m14.806407962s) [upload=478298]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 08:55:15.236234883 (545h26m15.107317928s) [upload=478298]: Done uploading segments... Dec 28 08:55:15.236259700 (545h26m15.107342745s) [upload=478298]: Upload finished. err: <nil> Duration: 30.003238267s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [....X.XX.XX.......XX.X....XXXX.XX....X....X.XX..XX..X.......] duration: 4.526471542s trace: https://jaeger.dev.storj.io/trace/77e06878e3b86651 FAIL (2025-12-28 08:55:29 UTC) orbiter warning: context deadline exceeded Duration: 4.526499043s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [.....X.XX.XX.......XX.X....XXXX.XX....X....X.XX..XX..X......] duration: 3.378390264s trace: https://jaeger.dev.storj.io/trace/77e06878e3b86651 FAIL (2025-12-28 08:54:50 UTC) orbiter warning: context deadline exceeded Duration: 3.378407477s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 8.178513427s trace: https://jaeger.dev.storj.io/trace/177689f80539b862 FAIL (2025-12-28 08:54:41 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 8.178542441s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X..X.XX.X.XX..X..XX....X...X.XX.XX.......X..X......XXXX.....] duration: 5.742889161s trace: https://jaeger.dev.storj.io/trace/667a4aac6d050408 FAIL (2025-12-28 08:54:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: YBXC3CEXJP3OTBYU3FMIMJSK4LPETZX55D3Z7VPR6NWY4YSEMBEA) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: OZOKGP6JN7CL6PJLWXDPG3QA2C5P5CMPMDDYOY2TVJMPKELNSBQQ) context deadline exceeded ======LOGS====== 08:50:10.285593840 (545h21m10.156676885s) [upload=478224]: Starting upload Dec 28 08:50:10.285688090 (545h21m10.156771155s) [upload=478224]: Done starting upload Dec 28 08:50:10.285836021 (545h21m10.156919056s) [upload=478224]: Uploading segments... Dec 28 08:50:10.285962142 (545h21m10.157045177s) [upload=478224]: Got next segment. Inline: false Dec 28 08:50:10.286117087 (545h21m10.157200132s) [upload=478224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 08:50:10.286124731 (545h21m10.157207776s) [upload=478224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 08:50:10.530460730 (545h21m10.401543775s) [seg_pos={0 0}, upload=478224]: Begin upload segment... Dec 28 08:50:10.530525243 (545h21m10.401608288s) [seg_pos={0 0}, upload=478224]: Done begin upload segment. Dec 28 08:50:10.530606627 (545h21m10.401689662s) [noise=true , address=77.90.31.19:4844 , piece_id=YBXC3CEXJP3OTBYU3FMIMJSK4LPETZX55D3Z7VPR6NWY4YSEMBEA, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.530634621 (545h21m10.401717666s) [noise=true , address=145.79.227.39:4013 , piece_id=JP34XIE6QZG2HJBONLUDACPYKCDTZQZW35LP2YC2VNBEBMDDWB3Q, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.531201670 (545h21m10.402284715s) [noise=true , address=145.79.227.37:4566 , piece_id=46FHYBKJ7KKE7XJSIYA7S5R23ZKB34UDUUV7PGTY7E2SJAS24SZQ, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.531776175 (545h21m10.40285922s) [noise=true , address=45.91.116.6:4975 , piece_id=QEREQPQV6F373QXQWDPQTBKW3BBOBZOB5FJ2XQWFY3BIPNV6JEDQ, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.532188299 (545h21m10.403271344s) [noise=true , address=45.91.116.8:4046 , piece_id=HVBKKQ4C75MYGYO42HFWI2RU35WC4RSX7BOTVTCXNXSRBMT22VOA, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.532624019 (545h21m10.403707054s) [noise=true , address=145.79.227.43:4159 , piece_id=IKFEA6LC4UNKJAR5EJTH5TNOTHAYXO34U5XWUTIP2MLLUDNDS6FA, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.533073034 (545h21m10.404156079s) [noise=true , address=145.79.227.48:4302 , piece_id=5C5N5KZV6QVPZFA6AZEBD2NOG7EJQ5XL437I6MKH2MOEJRGZCZ4Q, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.533540063 (545h21m10.404623108s) [noise=true , address=145.79.227.42:4121 , piece_id=WHDYTPU6FIBPH754UPZYQHV4AE5AH4S5X2UXU4IYLTOX62LSPSIA, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.535227467 (545h21m10.406310512s) [noise=true , address=145.79.227.17:4970 , piece_id=EBKOHG6YB3CKCAWAIBS6KW6NBTPIE2T5LOB4O3U2UX2V7GOVGCUA, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.536932173 (545h21m10.408015218s) [noise=true , address=145.79.227.11:4729 , piece_id=6KUUB7QM65FXIN2V6MNJJFC5B2PYV4MYUC4XB2CI6YMWU7CIWQ5Q, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.538421599 (545h21m10.409504644s) [noise=true , address=145.79.227.15:4873 , piece_id=ZZ2Y6VTARGFW4ROQOIA7SD34RWJRZGVTSN6NYYBWCMXIB4RBTBNQ, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.539724079 (545h21m10.410807124s) [noise=true , address=145.79.227.6:4259 , piece_id=CRXBD7264VZQLYJDJUPHKKJGFFWGDWYFUBW5V3PFENIR6KHRJIGA, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.540913543 (545h21m10.411996589s) [noise=true , address=145.79.227.13:4804 , piece_id=7RYCAZVU6BWZDBDJBSJ6CL57HY2EIY52RMII7QPC2LGWSR6YMXTQ, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.542259998 (545h21m10.413343033s) [noise=true , address=45.91.116.10:4084 , piece_id=OZOKGP6JN7CL6PJLWXDPG3QA2C5P5CMPMDDYOY2TVJMPKELNSBQQ, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.543536067 (545h21m10.414619102s) [noise=true , address=145.79.227.47:4265 , piece_id=JGPSKTCYBXM2WR2PQNYYSTGEESJIRWAMF5PFLH66XSU752XUF7RQ, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.547930976 (545h21m10.419014011s) [noise=true , address=45.91.116.5:4298 , piece_id=MAUITCSTYF5J6PHOWI2PICOYY36RAG4DH7OVP7TF3JXYPP3T2YJA, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:12.223866421 (545h21m12.094949466s) [upload=478224]: Next returned nil segment Dec 28 08:50:12.223883273 (545h21m12.094966308s) [upload=478224]: Waiting for error group managing segments... Dec 28 08:50:12.539177488 (545h21m12.410260523s) [noise=true , address=145.79.227.13:4804 , piece_id=7RYCAZVU6BWZDBDJBSJ6CL57HY2EIY52RMII7QPC2LGWSR6YMXTQ, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.549298942 (545h21m12.420382007s) [noise=true , address=145.79.227.48:4302 , piece_id=5C5N5KZV6QVPZFA6AZEBD2NOG7EJQ5XL437I6MKH2MOEJRGZCZ4Q, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.553637723 (545h21m12.424720758s) [noise=true , address=145.79.227.39:4013 , piece_id=JP34XIE6QZG2HJBONLUDACPYKCDTZQZW35LP2YC2VNBEBMDDWB3Q, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.553678732 (545h21m12.424761857s) [noise=true , address=145.79.227.42:4121 , piece_id=WHDYTPU6FIBPH754UPZYQHV4AE5AH4S5X2UXU4IYLTOX62LSPSIA, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.554408450 (545h21m12.425491495s) [noise=true , address=145.79.227.6:4259 , piece_id=CRXBD7264VZQLYJDJUPHKKJGFFWGDWYFUBW5V3PFENIR6KHRJIGA, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.555001991 (545h21m12.426085026s) [noise=true , address=145.79.227.47:4265 , piece_id=JGPSKTCYBXM2WR2PQNYYSTGEESJIRWAMF5PFLH66XSU752XUF7RQ, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.555157477 (545h21m12.426240522s) [noise=true , address=145.79.227.17:4970 , piece_id=EBKOHG6YB3CKCAWAIBS6KW6NBTPIE2T5LOB4O3U2UX2V7GOVGCUA, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.567732742 (545h21m12.438815787s) [noise=true , address=145.79.227.15:4873 , piece_id=ZZ2Y6VTARGFW4ROQOIA7SD34RWJRZGVTSN6NYYBWCMXIB4RBTBNQ, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.569506931 (545h21m12.440589976s) [noise=true , address=145.79.227.37:4566 , piece_id=46FHYBKJ7KKE7XJSIYA7S5R23ZKB34UDUUV7PGTY7E2SJAS24SZQ, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.570004098 (545h21m12.441087143s) [noise=true , address=145.79.227.11:4729 , piece_id=6KUUB7QM65FXIN2V6MNJJFC5B2PYV4MYUC4XB2CI6YMWU7CIWQ5Q, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.805556120 (545h21m12.676639165s) [noise=true , address=77.90.31.19:4844 , piece_id=YBXC3CEXJP3OTBYU3FMIMJSK4LPETZX55D3Z7VPR6NWY4YSEMBEA, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.823831730 (545h21m12.694914775s) [noise=true , address=45.91.116.10:4084 , piece_id=OZOKGP6JN7CL6PJLWXDPG3QA2C5P5CMPMDDYOY2TVJMPKELNSBQQ, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.925559107 (545h21m12.796642152s) [noise=true , address=45.91.116.6:4975 , piece_id=QEREQPQV6F373QXQWDPQTBKW3BBOBZOB5FJ2XQWFY3BIPNV6JEDQ, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.925585126 (545h21m12.796668181s) [seg_pos={0 0}, upload=478224]: Segment reached optimal threshold of 13 pieces. Dec 28 08:50:12.925969327 (545h21m12.797052373s) [noise=true , address=145.79.227.43:4159 , piece_id=IKFEA6LC4UNKJAR5EJTH5TNOTHAYXO34U5XWUTIP2MLLUDNDS6FA, seg_pos={0 0}, upload=478224]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 28 08:50:12.925979888 (545h21m12.797062933s) [noise=true , address=145.79.227.43:4159 , piece_id=IKFEA6LC4UNKJAR5EJTH5TNOTHAYXO34U5XWUTIP2MLLUDNDS6FA, seg_pos={0 0}, upload=478224]: Long tail cancellation due to optimal threshold Dec 28 08:50:12.926202612 (545h21m12.797285667s) [noise=true , address=45.91.116.8:4046 , piece_id=HVBKKQ4C75MYGYO42HFWI2RU35WC4RSX7BOTVTCXNXSRBMT22VOA, seg_pos={0 0}, upload=478224]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: write tcp 188.166.225.170:46342->45.91.116.8:4046: use of closed network connection; write tcp 188.166.225.170:46342->45.91.116.8:4046: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46342->45.91.116.8:4046: use of closed network connection Dec 28 08:50:12.926224965 (545h21m12.79730801s) [noise=true , address=45.91.116.8:4046 , piece_id=HVBKKQ4C75MYGYO42HFWI2RU35WC4RSX7BOTVTCXNXSRBMT22VOA, seg_pos={0 0}, upload=478224]: Long tail cancellation due to optimal threshold Dec 28 08:50:12.926505349 (545h21m12.797588384s) [noise=true , address=45.91.116.5:4298 , piece_id=MAUITCSTYF5J6PHOWI2PICOYY36RAG4DH7OVP7TF3JXYPP3T2YJA, seg_pos={0 0}, upload=478224]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 28 08:50:12.926513163 (545h21m12.797596198s) [noise=true , address=45.91.116.5:4298 , piece_id=MAUITCSTYF5J6PHOWI2PICOYY36RAG4DH7OVP7TF3JXYPP3T2YJA, seg_pos={0 0}, upload=478224]: Long tail cancellation due to optimal threshold Dec 28 08:50:12.926555704 (545h21m12.797638749s) [seg_pos={0 0}, upload=478224]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 08:50:12.926696422 (545h21m12.797779477s) [upload=478224]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 08:50:12.926721089 (545h21m12.797804144s) [upload=478224]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 08:50:13.223494458 (545h21m13.094577503s) [upload=478224]: Done uploading segments... Dec 28 08:50:13.223516159 (545h21m13.094599205s) [upload=478224]: Upload finished. err: <nil> Duration: 5.742917766s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [......X.XX.XX.......XX.X....XXXX.XX....X....X.XX..XX..X.....] duration: 3.584320011s trace: https://jaeger.dev.storj.io/trace/77e06878e3b86651 FAIL (2025-12-28 08:54:06 UTC) orbiter warning: context deadline exceeded Duration: 3.584346471s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 6.321183556s trace: https://jaeger.dev.storj.io/trace/177689f80539b862 FAIL (2025-12-28 08:54:04 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 6.321220755s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 6.983410231s trace: https://jaeger.dev.storj.io/trace/177689f80539b862 FAIL (2025-12-28 08:53:45 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 6.983431762s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 9.283765197s trace: https://jaeger.dev.storj.io/trace/177689f80539b862 FAIL (2025-12-28 08:53:28 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 9.283839037s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [XX..X.XX.X.XX..X..XX....X...X.XX.XX.......X..X......XXXX....] duration: 5.92181564s trace: https://jaeger.dev.storj.io/trace/667a4aac6d050408 FAIL (2025-12-28 08:53:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: YBXC3CEXJP3OTBYU3FMIMJSK4LPETZX55D3Z7VPR6NWY4YSEMBEA) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: OZOKGP6JN7CL6PJLWXDPG3QA2C5P5CMPMDDYOY2TVJMPKELNSBQQ) context deadline exceeded ======LOGS====== 08:50:10.285593840 (545h21m10.156676885s) [upload=478224]: Starting upload Dec 28 08:50:10.285688090 (545h21m10.156771155s) [upload=478224]: Done starting upload Dec 28 08:50:10.285836021 (545h21m10.156919056s) [upload=478224]: Uploading segments... Dec 28 08:50:10.285962142 (545h21m10.157045177s) [upload=478224]: Got next segment. Inline: false Dec 28 08:50:10.286117087 (545h21m10.157200132s) [upload=478224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 08:50:10.286124731 (545h21m10.157207776s) [upload=478224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 08:50:10.530460730 (545h21m10.401543775s) [seg_pos={0 0}, upload=478224]: Begin upload segment... Dec 28 08:50:10.530525243 (545h21m10.401608288s) [seg_pos={0 0}, upload=478224]: Done begin upload segment. Dec 28 08:50:10.530606627 (545h21m10.401689662s) [noise=true , address=77.90.31.19:4844 , piece_id=YBXC3CEXJP3OTBYU3FMIMJSK4LPETZX55D3Z7VPR6NWY4YSEMBEA, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.530634621 (545h21m10.401717666s) [noise=true , address=145.79.227.39:4013 , piece_id=JP34XIE6QZG2HJBONLUDACPYKCDTZQZW35LP2YC2VNBEBMDDWB3Q, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.531201670 (545h21m10.402284715s) [noise=true , address=145.79.227.37:4566 , piece_id=46FHYBKJ7KKE7XJSIYA7S5R23ZKB34UDUUV7PGTY7E2SJAS24SZQ, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.531776175 (545h21m10.40285922s) [noise=true , address=45.91.116.6:4975 , piece_id=QEREQPQV6F373QXQWDPQTBKW3BBOBZOB5FJ2XQWFY3BIPNV6JEDQ, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.532188299 (545h21m10.403271344s) [noise=true , address=45.91.116.8:4046 , piece_id=HVBKKQ4C75MYGYO42HFWI2RU35WC4RSX7BOTVTCXNXSRBMT22VOA, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.532624019 (545h21m10.403707054s) [noise=true , address=145.79.227.43:4159 , piece_id=IKFEA6LC4UNKJAR5EJTH5TNOTHAYXO34U5XWUTIP2MLLUDNDS6FA, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.533073034 (545h21m10.404156079s) [noise=true , address=145.79.227.48:4302 , piece_id=5C5N5KZV6QVPZFA6AZEBD2NOG7EJQ5XL437I6MKH2MOEJRGZCZ4Q, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.533540063 (545h21m10.404623108s) [noise=true , address=145.79.227.42:4121 , piece_id=WHDYTPU6FIBPH754UPZYQHV4AE5AH4S5X2UXU4IYLTOX62LSPSIA, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.535227467 (545h21m10.406310512s) [noise=true , address=145.79.227.17:4970 , piece_id=EBKOHG6YB3CKCAWAIBS6KW6NBTPIE2T5LOB4O3U2UX2V7GOVGCUA, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.536932173 (545h21m10.408015218s) [noise=true , address=145.79.227.11:4729 , piece_id=6KUUB7QM65FXIN2V6MNJJFC5B2PYV4MYUC4XB2CI6YMWU7CIWQ5Q, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.538421599 (545h21m10.409504644s) [noise=true , address=145.79.227.15:4873 , piece_id=ZZ2Y6VTARGFW4ROQOIA7SD34RWJRZGVTSN6NYYBWCMXIB4RBTBNQ, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.539724079 (545h21m10.410807124s) [noise=true , address=145.79.227.6:4259 , piece_id=CRXBD7264VZQLYJDJUPHKKJGFFWGDWYFUBW5V3PFENIR6KHRJIGA, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.540913543 (545h21m10.411996589s) [noise=true , address=145.79.227.13:4804 , piece_id=7RYCAZVU6BWZDBDJBSJ6CL57HY2EIY52RMII7QPC2LGWSR6YMXTQ, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.542259998 (545h21m10.413343033s) [noise=true , address=45.91.116.10:4084 , piece_id=OZOKGP6JN7CL6PJLWXDPG3QA2C5P5CMPMDDYOY2TVJMPKELNSBQQ, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.543536067 (545h21m10.414619102s) [noise=true , address=145.79.227.47:4265 , piece_id=JGPSKTCYBXM2WR2PQNYYSTGEESJIRWAMF5PFLH66XSU752XUF7RQ, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.547930976 (545h21m10.419014011s) [noise=true , address=45.91.116.5:4298 , piece_id=MAUITCSTYF5J6PHOWI2PICOYY36RAG4DH7OVP7TF3JXYPP3T2YJA, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:12.223866421 (545h21m12.094949466s) [upload=478224]: Next returned nil segment Dec 28 08:50:12.223883273 (545h21m12.094966308s) [upload=478224]: Waiting for error group managing segments... Dec 28 08:50:12.539177488 (545h21m12.410260523s) [noise=true , address=145.79.227.13:4804 , piece_id=7RYCAZVU6BWZDBDJBSJ6CL57HY2EIY52RMII7QPC2LGWSR6YMXTQ, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.549298942 (545h21m12.420382007s) [noise=true , address=145.79.227.48:4302 , piece_id=5C5N5KZV6QVPZFA6AZEBD2NOG7EJQ5XL437I6MKH2MOEJRGZCZ4Q, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.553637723 (545h21m12.424720758s) [noise=true , address=145.79.227.39:4013 , piece_id=JP34XIE6QZG2HJBONLUDACPYKCDTZQZW35LP2YC2VNBEBMDDWB3Q, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.553678732 (545h21m12.424761857s) [noise=true , address=145.79.227.42:4121 , piece_id=WHDYTPU6FIBPH754UPZYQHV4AE5AH4S5X2UXU4IYLTOX62LSPSIA, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.554408450 (545h21m12.425491495s) [noise=true , address=145.79.227.6:4259 , piece_id=CRXBD7264VZQLYJDJUPHKKJGFFWGDWYFUBW5V3PFENIR6KHRJIGA, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.555001991 (545h21m12.426085026s) [noise=true , address=145.79.227.47:4265 , piece_id=JGPSKTCYBXM2WR2PQNYYSTGEESJIRWAMF5PFLH66XSU752XUF7RQ, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.555157477 (545h21m12.426240522s) [noise=true , address=145.79.227.17:4970 , piece_id=EBKOHG6YB3CKCAWAIBS6KW6NBTPIE2T5LOB4O3U2UX2V7GOVGCUA, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.567732742 (545h21m12.438815787s) [noise=true , address=145.79.227.15:4873 , piece_id=ZZ2Y6VTARGFW4ROQOIA7SD34RWJRZGVTSN6NYYBWCMXIB4RBTBNQ, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.569506931 (545h21m12.440589976s) [noise=true , address=145.79.227.37:4566 , piece_id=46FHYBKJ7KKE7XJSIYA7S5R23ZKB34UDUUV7PGTY7E2SJAS24SZQ, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.570004098 (545h21m12.441087143s) [noise=true , address=145.79.227.11:4729 , piece_id=6KUUB7QM65FXIN2V6MNJJFC5B2PYV4MYUC4XB2CI6YMWU7CIWQ5Q, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.805556120 (545h21m12.676639165s) [noise=true , address=77.90.31.19:4844 , piece_id=YBXC3CEXJP3OTBYU3FMIMJSK4LPETZX55D3Z7VPR6NWY4YSEMBEA, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.823831730 (545h21m12.694914775s) [noise=true , address=45.91.116.10:4084 , piece_id=OZOKGP6JN7CL6PJLWXDPG3QA2C5P5CMPMDDYOY2TVJMPKELNSBQQ, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.925559107 (545h21m12.796642152s) [noise=true , address=45.91.116.6:4975 , piece_id=QEREQPQV6F373QXQWDPQTBKW3BBOBZOB5FJ2XQWFY3BIPNV6JEDQ, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.925585126 (545h21m12.796668181s) [seg_pos={0 0}, upload=478224]: Segment reached optimal threshold of 13 pieces. Dec 28 08:50:12.925969327 (545h21m12.797052373s) [noise=true , address=145.79.227.43:4159 , piece_id=IKFEA6LC4UNKJAR5EJTH5TNOTHAYXO34U5XWUTIP2MLLUDNDS6FA, seg_pos={0 0}, upload=478224]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 28 08:50:12.925979888 (545h21m12.797062933s) [noise=true , address=145.79.227.43:4159 , piece_id=IKFEA6LC4UNKJAR5EJTH5TNOTHAYXO34U5XWUTIP2MLLUDNDS6FA, seg_pos={0 0}, upload=478224]: Long tail cancellation due to optimal threshold Dec 28 08:50:12.926202612 (545h21m12.797285667s) [noise=true , address=45.91.116.8:4046 , piece_id=HVBKKQ4C75MYGYO42HFWI2RU35WC4RSX7BOTVTCXNXSRBMT22VOA, seg_pos={0 0}, upload=478224]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: write tcp 188.166.225.170:46342->45.91.116.8:4046: use of closed network connection; write tcp 188.166.225.170:46342->45.91.116.8:4046: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46342->45.91.116.8:4046: use of closed network connection Dec 28 08:50:12.926224965 (545h21m12.79730801s) [noise=true , address=45.91.116.8:4046 , piece_id=HVBKKQ4C75MYGYO42HFWI2RU35WC4RSX7BOTVTCXNXSRBMT22VOA, seg_pos={0 0}, upload=478224]: Long tail cancellation due to optimal threshold Dec 28 08:50:12.926505349 (545h21m12.797588384s) [noise=true , address=45.91.116.5:4298 , piece_id=MAUITCSTYF5J6PHOWI2PICOYY36RAG4DH7OVP7TF3JXYPP3T2YJA, seg_pos={0 0}, upload=478224]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 28 08:50:12.926513163 (545h21m12.797596198s) [noise=true , address=45.91.116.5:4298 , piece_id=MAUITCSTYF5J6PHOWI2PICOYY36RAG4DH7OVP7TF3JXYPP3T2YJA, seg_pos={0 0}, upload=478224]: Long tail cancellation due to optimal threshold Dec 28 08:50:12.926555704 (545h21m12.797638749s) [seg_pos={0 0}, upload=478224]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 08:50:12.926696422 (545h21m12.797779477s) [upload=478224]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 08:50:12.926721089 (545h21m12.797804144s) [upload=478224]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 08:50:13.223494458 (545h21m13.094577503s) [upload=478224]: Done uploading segments... Dec 28 08:50:13.223516159 (545h21m13.094599205s) [upload=478224]: Upload finished. err: <nil> Duration: 5.921851347s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 21 out of 60 is failed (threshold=6) [.......X.XX.XX.......XX.X....XXXX.XX....X....X.XX..XX..X....] duration: 4.750079769s trace: https://jaeger.dev.storj.io/trace/77e06878e3b86651 FAIL (2025-12-28 08:53:02 UTC) orbiter warning: context deadline exceeded Duration: 4.750104516s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 22 out of 60 is failed (threshold=6) [X.......X.XX.XX.......XX.X....XXXX.XX....X....X.XX..XX..X...] duration: 3.593535047s trace: https://jaeger.dev.storj.io/trace/77e06878e3b86651 FAIL (2025-12-28 08:52:37 UTC) orbiter warning: context deadline exceeded Duration: 3.593565164s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [XXX..X.XX.X.XX..X..XX....X...X.XX.XX.......X..X......XXXX...] duration: 5.982266379s trace: https://jaeger.dev.storj.io/trace/667a4aac6d050408 FAIL (2025-12-28 08:52:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: YBXC3CEXJP3OTBYU3FMIMJSK4LPETZX55D3Z7VPR6NWY4YSEMBEA) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: OZOKGP6JN7CL6PJLWXDPG3QA2C5P5CMPMDDYOY2TVJMPKELNSBQQ) context deadline exceeded ======LOGS====== 08:50:10.285593840 (545h21m10.156676885s) [upload=478224]: Starting upload Dec 28 08:50:10.285688090 (545h21m10.156771155s) [upload=478224]: Done starting upload Dec 28 08:50:10.285836021 (545h21m10.156919056s) [upload=478224]: Uploading segments... Dec 28 08:50:10.285962142 (545h21m10.157045177s) [upload=478224]: Got next segment. Inline: false Dec 28 08:50:10.286117087 (545h21m10.157200132s) [upload=478224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 08:50:10.286124731 (545h21m10.157207776s) [upload=478224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 08:50:10.530460730 (545h21m10.401543775s) [seg_pos={0 0}, upload=478224]: Begin upload segment... Dec 28 08:50:10.530525243 (545h21m10.401608288s) [seg_pos={0 0}, upload=478224]: Done begin upload segment. Dec 28 08:50:10.530606627 (545h21m10.401689662s) [noise=true , address=77.90.31.19:4844 , piece_id=YBXC3CEXJP3OTBYU3FMIMJSK4LPETZX55D3Z7VPR6NWY4YSEMBEA, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.530634621 (545h21m10.401717666s) [noise=true , address=145.79.227.39:4013 , piece_id=JP34XIE6QZG2HJBONLUDACPYKCDTZQZW35LP2YC2VNBEBMDDWB3Q, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.531201670 (545h21m10.402284715s) [noise=true , address=145.79.227.37:4566 , piece_id=46FHYBKJ7KKE7XJSIYA7S5R23ZKB34UDUUV7PGTY7E2SJAS24SZQ, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.531776175 (545h21m10.40285922s) [noise=true , address=45.91.116.6:4975 , piece_id=QEREQPQV6F373QXQWDPQTBKW3BBOBZOB5FJ2XQWFY3BIPNV6JEDQ, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.532188299 (545h21m10.403271344s) [noise=true , address=45.91.116.8:4046 , piece_id=HVBKKQ4C75MYGYO42HFWI2RU35WC4RSX7BOTVTCXNXSRBMT22VOA, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.532624019 (545h21m10.403707054s) [noise=true , address=145.79.227.43:4159 , piece_id=IKFEA6LC4UNKJAR5EJTH5TNOTHAYXO34U5XWUTIP2MLLUDNDS6FA, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.533073034 (545h21m10.404156079s) [noise=true , address=145.79.227.48:4302 , piece_id=5C5N5KZV6QVPZFA6AZEBD2NOG7EJQ5XL437I6MKH2MOEJRGZCZ4Q, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.533540063 (545h21m10.404623108s) [noise=true , address=145.79.227.42:4121 , piece_id=WHDYTPU6FIBPH754UPZYQHV4AE5AH4S5X2UXU4IYLTOX62LSPSIA, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.535227467 (545h21m10.406310512s) [noise=true , address=145.79.227.17:4970 , piece_id=EBKOHG6YB3CKCAWAIBS6KW6NBTPIE2T5LOB4O3U2UX2V7GOVGCUA, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.536932173 (545h21m10.408015218s) [noise=true , address=145.79.227.11:4729 , piece_id=6KUUB7QM65FXIN2V6MNJJFC5B2PYV4MYUC4XB2CI6YMWU7CIWQ5Q, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.538421599 (545h21m10.409504644s) [noise=true , address=145.79.227.15:4873 , piece_id=ZZ2Y6VTARGFW4ROQOIA7SD34RWJRZGVTSN6NYYBWCMXIB4RBTBNQ, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.539724079 (545h21m10.410807124s) [noise=true , address=145.79.227.6:4259 , piece_id=CRXBD7264VZQLYJDJUPHKKJGFFWGDWYFUBW5V3PFENIR6KHRJIGA, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.540913543 (545h21m10.411996589s) [noise=true , address=145.79.227.13:4804 , piece_id=7RYCAZVU6BWZDBDJBSJ6CL57HY2EIY52RMII7QPC2LGWSR6YMXTQ, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.542259998 (545h21m10.413343033s) [noise=true , address=45.91.116.10:4084 , piece_id=OZOKGP6JN7CL6PJLWXDPG3QA2C5P5CMPMDDYOY2TVJMPKELNSBQQ, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.543536067 (545h21m10.414619102s) [noise=true , address=145.79.227.47:4265 , piece_id=JGPSKTCYBXM2WR2PQNYYSTGEESJIRWAMF5PFLH66XSU752XUF7RQ, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.547930976 (545h21m10.419014011s) [noise=true , address=45.91.116.5:4298 , piece_id=MAUITCSTYF5J6PHOWI2PICOYY36RAG4DH7OVP7TF3JXYPP3T2YJA, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:12.223866421 (545h21m12.094949466s) [upload=478224]: Next returned nil segment Dec 28 08:50:12.223883273 (545h21m12.094966308s) [upload=478224]: Waiting for error group managing segments... Dec 28 08:50:12.539177488 (545h21m12.410260523s) [noise=true , address=145.79.227.13:4804 , piece_id=7RYCAZVU6BWZDBDJBSJ6CL57HY2EIY52RMII7QPC2LGWSR6YMXTQ, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.549298942 (545h21m12.420382007s) [noise=true , address=145.79.227.48:4302 , piece_id=5C5N5KZV6QVPZFA6AZEBD2NOG7EJQ5XL437I6MKH2MOEJRGZCZ4Q, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.553637723 (545h21m12.424720758s) [noise=true , address=145.79.227.39:4013 , piece_id=JP34XIE6QZG2HJBONLUDACPYKCDTZQZW35LP2YC2VNBEBMDDWB3Q, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.553678732 (545h21m12.424761857s) [noise=true , address=145.79.227.42:4121 , piece_id=WHDYTPU6FIBPH754UPZYQHV4AE5AH4S5X2UXU4IYLTOX62LSPSIA, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.554408450 (545h21m12.425491495s) [noise=true , address=145.79.227.6:4259 , piece_id=CRXBD7264VZQLYJDJUPHKKJGFFWGDWYFUBW5V3PFENIR6KHRJIGA, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.555001991 (545h21m12.426085026s) [noise=true , address=145.79.227.47:4265 , piece_id=JGPSKTCYBXM2WR2PQNYYSTGEESJIRWAMF5PFLH66XSU752XUF7RQ, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.555157477 (545h21m12.426240522s) [noise=true , address=145.79.227.17:4970 , piece_id=EBKOHG6YB3CKCAWAIBS6KW6NBTPIE2T5LOB4O3U2UX2V7GOVGCUA, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.567732742 (545h21m12.438815787s) [noise=true , address=145.79.227.15:4873 , piece_id=ZZ2Y6VTARGFW4ROQOIA7SD34RWJRZGVTSN6NYYBWCMXIB4RBTBNQ, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.569506931 (545h21m12.440589976s) [noise=true , address=145.79.227.37:4566 , piece_id=46FHYBKJ7KKE7XJSIYA7S5R23ZKB34UDUUV7PGTY7E2SJAS24SZQ, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.570004098 (545h21m12.441087143s) [noise=true , address=145.79.227.11:4729 , piece_id=6KUUB7QM65FXIN2V6MNJJFC5B2PYV4MYUC4XB2CI6YMWU7CIWQ5Q, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.805556120 (545h21m12.676639165s) [noise=true , address=77.90.31.19:4844 , piece_id=YBXC3CEXJP3OTBYU3FMIMJSK4LPETZX55D3Z7VPR6NWY4YSEMBEA, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.823831730 (545h21m12.694914775s) [noise=true , address=45.91.116.10:4084 , piece_id=OZOKGP6JN7CL6PJLWXDPG3QA2C5P5CMPMDDYOY2TVJMPKELNSBQQ, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.925559107 (545h21m12.796642152s) [noise=true , address=45.91.116.6:4975 , piece_id=QEREQPQV6F373QXQWDPQTBKW3BBOBZOB5FJ2XQWFY3BIPNV6JEDQ, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.925585126 (545h21m12.796668181s) [seg_pos={0 0}, upload=478224]: Segment reached optimal threshold of 13 pieces. Dec 28 08:50:12.925969327 (545h21m12.797052373s) [noise=true , address=145.79.227.43:4159 , piece_id=IKFEA6LC4UNKJAR5EJTH5TNOTHAYXO34U5XWUTIP2MLLUDNDS6FA, seg_pos={0 0}, upload=478224]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 28 08:50:12.925979888 (545h21m12.797062933s) [noise=true , address=145.79.227.43:4159 , piece_id=IKFEA6LC4UNKJAR5EJTH5TNOTHAYXO34U5XWUTIP2MLLUDNDS6FA, seg_pos={0 0}, upload=478224]: Long tail cancellation due to optimal threshold Dec 28 08:50:12.926202612 (545h21m12.797285667s) [noise=true , address=45.91.116.8:4046 , piece_id=HVBKKQ4C75MYGYO42HFWI2RU35WC4RSX7BOTVTCXNXSRBMT22VOA, seg_pos={0 0}, upload=478224]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: write tcp 188.166.225.170:46342->45.91.116.8:4046: use of closed network connection; write tcp 188.166.225.170:46342->45.91.116.8:4046: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46342->45.91.116.8:4046: use of closed network connection Dec 28 08:50:12.926224965 (545h21m12.79730801s) [noise=true , address=45.91.116.8:4046 , piece_id=HVBKKQ4C75MYGYO42HFWI2RU35WC4RSX7BOTVTCXNXSRBMT22VOA, seg_pos={0 0}, upload=478224]: Long tail cancellation due to optimal threshold Dec 28 08:50:12.926505349 (545h21m12.797588384s) [noise=true , address=45.91.116.5:4298 , piece_id=MAUITCSTYF5J6PHOWI2PICOYY36RAG4DH7OVP7TF3JXYPP3T2YJA, seg_pos={0 0}, upload=478224]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 28 08:50:12.926513163 (545h21m12.797596198s) [noise=true , address=45.91.116.5:4298 , piece_id=MAUITCSTYF5J6PHOWI2PICOYY36RAG4DH7OVP7TF3JXYPP3T2YJA, seg_pos={0 0}, upload=478224]: Long tail cancellation due to optimal threshold Dec 28 08:50:12.926555704 (545h21m12.797638749s) [seg_pos={0 0}, upload=478224]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 08:50:12.926696422 (545h21m12.797779477s) [upload=478224]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 08:50:12.926721089 (545h21m12.797804144s) [upload=478224]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 08:50:13.223494458 (545h21m13.094577503s) [upload=478224]: Done uploading segments... Dec 28 08:50:13.223516159 (545h21m13.094599205s) [upload=478224]: Upload finished. err: <nil> Duration: 5.982296285s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 10.013495181s trace: https://jaeger.dev.storj.io/trace/177689f80539b862 FAIL (2025-12-28 08:52:33 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 10.013516722s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [XX.......X.XX.XX.......XX.X....XXXX.XX....X....X.XX..XX..X..] duration: 3.456693409s trace: https://jaeger.dev.storj.io/trace/77e06878e3b86651 FAIL (2025-12-28 08:51:56 UTC) orbiter warning: context deadline exceeded Duration: 3.456838707s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [.XXX..X.XX.X.XX..X..XX....X...X.XX.XX.......X..X......XXXX..] duration: 5.940019137s trace: https://jaeger.dev.storj.io/trace/667a4aac6d050408 FAIL (2025-12-28 08:51:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: YBXC3CEXJP3OTBYU3FMIMJSK4LPETZX55D3Z7VPR6NWY4YSEMBEA) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: OZOKGP6JN7CL6PJLWXDPG3QA2C5P5CMPMDDYOY2TVJMPKELNSBQQ) context deadline exceeded ======LOGS====== 08:50:10.285593840 (545h21m10.156676885s) [upload=478224]: Starting upload Dec 28 08:50:10.285688090 (545h21m10.156771155s) [upload=478224]: Done starting upload Dec 28 08:50:10.285836021 (545h21m10.156919056s) [upload=478224]: Uploading segments... Dec 28 08:50:10.285962142 (545h21m10.157045177s) [upload=478224]: Got next segment. Inline: false Dec 28 08:50:10.286117087 (545h21m10.157200132s) [upload=478224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 08:50:10.286124731 (545h21m10.157207776s) [upload=478224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 08:50:10.530460730 (545h21m10.401543775s) [seg_pos={0 0}, upload=478224]: Begin upload segment... Dec 28 08:50:10.530525243 (545h21m10.401608288s) [seg_pos={0 0}, upload=478224]: Done begin upload segment. Dec 28 08:50:10.530606627 (545h21m10.401689662s) [noise=true , address=77.90.31.19:4844 , piece_id=YBXC3CEXJP3OTBYU3FMIMJSK4LPETZX55D3Z7VPR6NWY4YSEMBEA, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.530634621 (545h21m10.401717666s) [noise=true , address=145.79.227.39:4013 , piece_id=JP34XIE6QZG2HJBONLUDACPYKCDTZQZW35LP2YC2VNBEBMDDWB3Q, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.531201670 (545h21m10.402284715s) [noise=true , address=145.79.227.37:4566 , piece_id=46FHYBKJ7KKE7XJSIYA7S5R23ZKB34UDUUV7PGTY7E2SJAS24SZQ, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.531776175 (545h21m10.40285922s) [noise=true , address=45.91.116.6:4975 , piece_id=QEREQPQV6F373QXQWDPQTBKW3BBOBZOB5FJ2XQWFY3BIPNV6JEDQ, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.532188299 (545h21m10.403271344s) [noise=true , address=45.91.116.8:4046 , piece_id=HVBKKQ4C75MYGYO42HFWI2RU35WC4RSX7BOTVTCXNXSRBMT22VOA, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.532624019 (545h21m10.403707054s) [noise=true , address=145.79.227.43:4159 , piece_id=IKFEA6LC4UNKJAR5EJTH5TNOTHAYXO34U5XWUTIP2MLLUDNDS6FA, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.533073034 (545h21m10.404156079s) [noise=true , address=145.79.227.48:4302 , piece_id=5C5N5KZV6QVPZFA6AZEBD2NOG7EJQ5XL437I6MKH2MOEJRGZCZ4Q, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.533540063 (545h21m10.404623108s) [noise=true , address=145.79.227.42:4121 , piece_id=WHDYTPU6FIBPH754UPZYQHV4AE5AH4S5X2UXU4IYLTOX62LSPSIA, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.535227467 (545h21m10.406310512s) [noise=true , address=145.79.227.17:4970 , piece_id=EBKOHG6YB3CKCAWAIBS6KW6NBTPIE2T5LOB4O3U2UX2V7GOVGCUA, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.536932173 (545h21m10.408015218s) [noise=true , address=145.79.227.11:4729 , piece_id=6KUUB7QM65FXIN2V6MNJJFC5B2PYV4MYUC4XB2CI6YMWU7CIWQ5Q, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.538421599 (545h21m10.409504644s) [noise=true , address=145.79.227.15:4873 , piece_id=ZZ2Y6VTARGFW4ROQOIA7SD34RWJRZGVTSN6NYYBWCMXIB4RBTBNQ, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.539724079 (545h21m10.410807124s) [noise=true , address=145.79.227.6:4259 , piece_id=CRXBD7264VZQLYJDJUPHKKJGFFWGDWYFUBW5V3PFENIR6KHRJIGA, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.540913543 (545h21m10.411996589s) [noise=true , address=145.79.227.13:4804 , piece_id=7RYCAZVU6BWZDBDJBSJ6CL57HY2EIY52RMII7QPC2LGWSR6YMXTQ, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.542259998 (545h21m10.413343033s) [noise=true , address=45.91.116.10:4084 , piece_id=OZOKGP6JN7CL6PJLWXDPG3QA2C5P5CMPMDDYOY2TVJMPKELNSBQQ, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.543536067 (545h21m10.414619102s) [noise=true , address=145.79.227.47:4265 , piece_id=JGPSKTCYBXM2WR2PQNYYSTGEESJIRWAMF5PFLH66XSU752XUF7RQ, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.547930976 (545h21m10.419014011s) [noise=true , address=45.91.116.5:4298 , piece_id=MAUITCSTYF5J6PHOWI2PICOYY36RAG4DH7OVP7TF3JXYPP3T2YJA, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:12.223866421 (545h21m12.094949466s) [upload=478224]: Next returned nil segment Dec 28 08:50:12.223883273 (545h21m12.094966308s) [upload=478224]: Waiting for error group managing segments... Dec 28 08:50:12.539177488 (545h21m12.410260523s) [noise=true , address=145.79.227.13:4804 , piece_id=7RYCAZVU6BWZDBDJBSJ6CL57HY2EIY52RMII7QPC2LGWSR6YMXTQ, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.549298942 (545h21m12.420382007s) [noise=true , address=145.79.227.48:4302 , piece_id=5C5N5KZV6QVPZFA6AZEBD2NOG7EJQ5XL437I6MKH2MOEJRGZCZ4Q, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.553637723 (545h21m12.424720758s) [noise=true , address=145.79.227.39:4013 , piece_id=JP34XIE6QZG2HJBONLUDACPYKCDTZQZW35LP2YC2VNBEBMDDWB3Q, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.553678732 (545h21m12.424761857s) [noise=true , address=145.79.227.42:4121 , piece_id=WHDYTPU6FIBPH754UPZYQHV4AE5AH4S5X2UXU4IYLTOX62LSPSIA, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.554408450 (545h21m12.425491495s) [noise=true , address=145.79.227.6:4259 , piece_id=CRXBD7264VZQLYJDJUPHKKJGFFWGDWYFUBW5V3PFENIR6KHRJIGA, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.555001991 (545h21m12.426085026s) [noise=true , address=145.79.227.47:4265 , piece_id=JGPSKTCYBXM2WR2PQNYYSTGEESJIRWAMF5PFLH66XSU752XUF7RQ, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.555157477 (545h21m12.426240522s) [noise=true , address=145.79.227.17:4970 , piece_id=EBKOHG6YB3CKCAWAIBS6KW6NBTPIE2T5LOB4O3U2UX2V7GOVGCUA, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.567732742 (545h21m12.438815787s) [noise=true , address=145.79.227.15:4873 , piece_id=ZZ2Y6VTARGFW4ROQOIA7SD34RWJRZGVTSN6NYYBWCMXIB4RBTBNQ, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.569506931 (545h21m12.440589976s) [noise=true , address=145.79.227.37:4566 , piece_id=46FHYBKJ7KKE7XJSIYA7S5R23ZKB34UDUUV7PGTY7E2SJAS24SZQ, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.570004098 (545h21m12.441087143s) [noise=true , address=145.79.227.11:4729 , piece_id=6KUUB7QM65FXIN2V6MNJJFC5B2PYV4MYUC4XB2CI6YMWU7CIWQ5Q, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.805556120 (545h21m12.676639165s) [noise=true , address=77.90.31.19:4844 , piece_id=YBXC3CEXJP3OTBYU3FMIMJSK4LPETZX55D3Z7VPR6NWY4YSEMBEA, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.823831730 (545h21m12.694914775s) [noise=true , address=45.91.116.10:4084 , piece_id=OZOKGP6JN7CL6PJLWXDPG3QA2C5P5CMPMDDYOY2TVJMPKELNSBQQ, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.925559107 (545h21m12.796642152s) [noise=true , address=45.91.116.6:4975 , piece_id=QEREQPQV6F373QXQWDPQTBKW3BBOBZOB5FJ2XQWFY3BIPNV6JEDQ, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.925585126 (545h21m12.796668181s) [seg_pos={0 0}, upload=478224]: Segment reached optimal threshold of 13 pieces. Dec 28 08:50:12.925969327 (545h21m12.797052373s) [noise=true , address=145.79.227.43:4159 , piece_id=IKFEA6LC4UNKJAR5EJTH5TNOTHAYXO34U5XWUTIP2MLLUDNDS6FA, seg_pos={0 0}, upload=478224]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 28 08:50:12.925979888 (545h21m12.797062933s) [noise=true , address=145.79.227.43:4159 , piece_id=IKFEA6LC4UNKJAR5EJTH5TNOTHAYXO34U5XWUTIP2MLLUDNDS6FA, seg_pos={0 0}, upload=478224]: Long tail cancellation due to optimal threshold Dec 28 08:50:12.926202612 (545h21m12.797285667s) [noise=true , address=45.91.116.8:4046 , piece_id=HVBKKQ4C75MYGYO42HFWI2RU35WC4RSX7BOTVTCXNXSRBMT22VOA, seg_pos={0 0}, upload=478224]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: write tcp 188.166.225.170:46342->45.91.116.8:4046: use of closed network connection; write tcp 188.166.225.170:46342->45.91.116.8:4046: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46342->45.91.116.8:4046: use of closed network connection Dec 28 08:50:12.926224965 (545h21m12.79730801s) [noise=true , address=45.91.116.8:4046 , piece_id=HVBKKQ4C75MYGYO42HFWI2RU35WC4RSX7BOTVTCXNXSRBMT22VOA, seg_pos={0 0}, upload=478224]: Long tail cancellation due to optimal threshold Dec 28 08:50:12.926505349 (545h21m12.797588384s) [noise=true , address=45.91.116.5:4298 , piece_id=MAUITCSTYF5J6PHOWI2PICOYY36RAG4DH7OVP7TF3JXYPP3T2YJA, seg_pos={0 0}, upload=478224]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 28 08:50:12.926513163 (545h21m12.797596198s) [noise=true , address=45.91.116.5:4298 , piece_id=MAUITCSTYF5J6PHOWI2PICOYY36RAG4DH7OVP7TF3JXYPP3T2YJA, seg_pos={0 0}, upload=478224]: Long tail cancellation due to optimal threshold Dec 28 08:50:12.926555704 (545h21m12.797638749s) [seg_pos={0 0}, upload=478224]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 08:50:12.926696422 (545h21m12.797779477s) [upload=478224]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 08:50:12.926721089 (545h21m12.797804144s) [upload=478224]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 08:50:13.223494458 (545h21m13.094577503s) [upload=478224]: Done uploading segments... Dec 28 08:50:13.223516159 (545h21m13.094599205s) [upload=478224]: Upload finished. err: <nil> Duration: 5.940051529s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.888533389s trace: https://jaeger.dev.storj.io/trace/177689f80539b862 FAIL (2025-12-28 08:51:40 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 5.888557636s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 23 out of 60 is failed (threshold=6) [.XX.......X.XX.XX.......XX.X....XXXX.XX....X....X.XX..XX..X.] duration: 3.457101506s trace: https://jaeger.dev.storj.io/trace/77e06878e3b86651 FAIL (2025-12-28 08:51:36 UTC) orbiter warning: context deadline exceeded Duration: 3.457123008s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [X.XXX..X.XX.X.XX..X..XX....X...X.XX.XX.......X..X......XXXX.] duration: 5.942220007s trace: https://jaeger.dev.storj.io/trace/667a4aac6d050408 FAIL (2025-12-28 08:51:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: YBXC3CEXJP3OTBYU3FMIMJSK4LPETZX55D3Z7VPR6NWY4YSEMBEA) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: OZOKGP6JN7CL6PJLWXDPG3QA2C5P5CMPMDDYOY2TVJMPKELNSBQQ) context deadline exceeded ======LOGS====== 08:50:10.285593840 (545h21m10.156676885s) [upload=478224]: Starting upload Dec 28 08:50:10.285688090 (545h21m10.156771155s) [upload=478224]: Done starting upload Dec 28 08:50:10.285836021 (545h21m10.156919056s) [upload=478224]: Uploading segments... Dec 28 08:50:10.285962142 (545h21m10.157045177s) [upload=478224]: Got next segment. Inline: false Dec 28 08:50:10.286117087 (545h21m10.157200132s) [upload=478224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 08:50:10.286124731 (545h21m10.157207776s) [upload=478224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 08:50:10.530460730 (545h21m10.401543775s) [seg_pos={0 0}, upload=478224]: Begin upload segment... Dec 28 08:50:10.530525243 (545h21m10.401608288s) [seg_pos={0 0}, upload=478224]: Done begin upload segment. Dec 28 08:50:10.530606627 (545h21m10.401689662s) [noise=true , address=77.90.31.19:4844 , piece_id=YBXC3CEXJP3OTBYU3FMIMJSK4LPETZX55D3Z7VPR6NWY4YSEMBEA, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.530634621 (545h21m10.401717666s) [noise=true , address=145.79.227.39:4013 , piece_id=JP34XIE6QZG2HJBONLUDACPYKCDTZQZW35LP2YC2VNBEBMDDWB3Q, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.531201670 (545h21m10.402284715s) [noise=true , address=145.79.227.37:4566 , piece_id=46FHYBKJ7KKE7XJSIYA7S5R23ZKB34UDUUV7PGTY7E2SJAS24SZQ, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.531776175 (545h21m10.40285922s) [noise=true , address=45.91.116.6:4975 , piece_id=QEREQPQV6F373QXQWDPQTBKW3BBOBZOB5FJ2XQWFY3BIPNV6JEDQ, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.532188299 (545h21m10.403271344s) [noise=true , address=45.91.116.8:4046 , piece_id=HVBKKQ4C75MYGYO42HFWI2RU35WC4RSX7BOTVTCXNXSRBMT22VOA, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.532624019 (545h21m10.403707054s) [noise=true , address=145.79.227.43:4159 , piece_id=IKFEA6LC4UNKJAR5EJTH5TNOTHAYXO34U5XWUTIP2MLLUDNDS6FA, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.533073034 (545h21m10.404156079s) [noise=true , address=145.79.227.48:4302 , piece_id=5C5N5KZV6QVPZFA6AZEBD2NOG7EJQ5XL437I6MKH2MOEJRGZCZ4Q, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.533540063 (545h21m10.404623108s) [noise=true , address=145.79.227.42:4121 , piece_id=WHDYTPU6FIBPH754UPZYQHV4AE5AH4S5X2UXU4IYLTOX62LSPSIA, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.535227467 (545h21m10.406310512s) [noise=true , address=145.79.227.17:4970 , piece_id=EBKOHG6YB3CKCAWAIBS6KW6NBTPIE2T5LOB4O3U2UX2V7GOVGCUA, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.536932173 (545h21m10.408015218s) [noise=true , address=145.79.227.11:4729 , piece_id=6KUUB7QM65FXIN2V6MNJJFC5B2PYV4MYUC4XB2CI6YMWU7CIWQ5Q, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.538421599 (545h21m10.409504644s) [noise=true , address=145.79.227.15:4873 , piece_id=ZZ2Y6VTARGFW4ROQOIA7SD34RWJRZGVTSN6NYYBWCMXIB4RBTBNQ, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.539724079 (545h21m10.410807124s) [noise=true , address=145.79.227.6:4259 , piece_id=CRXBD7264VZQLYJDJUPHKKJGFFWGDWYFUBW5V3PFENIR6KHRJIGA, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.540913543 (545h21m10.411996589s) [noise=true , address=145.79.227.13:4804 , piece_id=7RYCAZVU6BWZDBDJBSJ6CL57HY2EIY52RMII7QPC2LGWSR6YMXTQ, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.542259998 (545h21m10.413343033s) [noise=true , address=45.91.116.10:4084 , piece_id=OZOKGP6JN7CL6PJLWXDPG3QA2C5P5CMPMDDYOY2TVJMPKELNSBQQ, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.543536067 (545h21m10.414619102s) [noise=true , address=145.79.227.47:4265 , piece_id=JGPSKTCYBXM2WR2PQNYYSTGEESJIRWAMF5PFLH66XSU752XUF7RQ, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.547930976 (545h21m10.419014011s) [noise=true , address=45.91.116.5:4298 , piece_id=MAUITCSTYF5J6PHOWI2PICOYY36RAG4DH7OVP7TF3JXYPP3T2YJA, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:12.223866421 (545h21m12.094949466s) [upload=478224]: Next returned nil segment Dec 28 08:50:12.223883273 (545h21m12.094966308s) [upload=478224]: Waiting for error group managing segments... Dec 28 08:50:12.539177488 (545h21m12.410260523s) [noise=true , address=145.79.227.13:4804 , piece_id=7RYCAZVU6BWZDBDJBSJ6CL57HY2EIY52RMII7QPC2LGWSR6YMXTQ, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.549298942 (545h21m12.420382007s) [noise=true , address=145.79.227.48:4302 , piece_id=5C5N5KZV6QVPZFA6AZEBD2NOG7EJQ5XL437I6MKH2MOEJRGZCZ4Q, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.553637723 (545h21m12.424720758s) [noise=true , address=145.79.227.39:4013 , piece_id=JP34XIE6QZG2HJBONLUDACPYKCDTZQZW35LP2YC2VNBEBMDDWB3Q, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.553678732 (545h21m12.424761857s) [noise=true , address=145.79.227.42:4121 , piece_id=WHDYTPU6FIBPH754UPZYQHV4AE5AH4S5X2UXU4IYLTOX62LSPSIA, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.554408450 (545h21m12.425491495s) [noise=true , address=145.79.227.6:4259 , piece_id=CRXBD7264VZQLYJDJUPHKKJGFFWGDWYFUBW5V3PFENIR6KHRJIGA, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.555001991 (545h21m12.426085026s) [noise=true , address=145.79.227.47:4265 , piece_id=JGPSKTCYBXM2WR2PQNYYSTGEESJIRWAMF5PFLH66XSU752XUF7RQ, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.555157477 (545h21m12.426240522s) [noise=true , address=145.79.227.17:4970 , piece_id=EBKOHG6YB3CKCAWAIBS6KW6NBTPIE2T5LOB4O3U2UX2V7GOVGCUA, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.567732742 (545h21m12.438815787s) [noise=true , address=145.79.227.15:4873 , piece_id=ZZ2Y6VTARGFW4ROQOIA7SD34RWJRZGVTSN6NYYBWCMXIB4RBTBNQ, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.569506931 (545h21m12.440589976s) [noise=true , address=145.79.227.37:4566 , piece_id=46FHYBKJ7KKE7XJSIYA7S5R23ZKB34UDUUV7PGTY7E2SJAS24SZQ, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.570004098 (545h21m12.441087143s) [noise=true , address=145.79.227.11:4729 , piece_id=6KUUB7QM65FXIN2V6MNJJFC5B2PYV4MYUC4XB2CI6YMWU7CIWQ5Q, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.805556120 (545h21m12.676639165s) [noise=true , address=77.90.31.19:4844 , piece_id=YBXC3CEXJP3OTBYU3FMIMJSK4LPETZX55D3Z7VPR6NWY4YSEMBEA, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.823831730 (545h21m12.694914775s) [noise=true , address=45.91.116.10:4084 , piece_id=OZOKGP6JN7CL6PJLWXDPG3QA2C5P5CMPMDDYOY2TVJMPKELNSBQQ, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.925559107 (545h21m12.796642152s) [noise=true , address=45.91.116.6:4975 , piece_id=QEREQPQV6F373QXQWDPQTBKW3BBOBZOB5FJ2XQWFY3BIPNV6JEDQ, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.925585126 (545h21m12.796668181s) [seg_pos={0 0}, upload=478224]: Segment reached optimal threshold of 13 pieces. Dec 28 08:50:12.925969327 (545h21m12.797052373s) [noise=true , address=145.79.227.43:4159 , piece_id=IKFEA6LC4UNKJAR5EJTH5TNOTHAYXO34U5XWUTIP2MLLUDNDS6FA, seg_pos={0 0}, upload=478224]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 28 08:50:12.925979888 (545h21m12.797062933s) [noise=true , address=145.79.227.43:4159 , piece_id=IKFEA6LC4UNKJAR5EJTH5TNOTHAYXO34U5XWUTIP2MLLUDNDS6FA, seg_pos={0 0}, upload=478224]: Long tail cancellation due to optimal threshold Dec 28 08:50:12.926202612 (545h21m12.797285667s) [noise=true , address=45.91.116.8:4046 , piece_id=HVBKKQ4C75MYGYO42HFWI2RU35WC4RSX7BOTVTCXNXSRBMT22VOA, seg_pos={0 0}, upload=478224]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: write tcp 188.166.225.170:46342->45.91.116.8:4046: use of closed network connection; write tcp 188.166.225.170:46342->45.91.116.8:4046: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46342->45.91.116.8:4046: use of closed network connection Dec 28 08:50:12.926224965 (545h21m12.79730801s) [noise=true , address=45.91.116.8:4046 , piece_id=HVBKKQ4C75MYGYO42HFWI2RU35WC4RSX7BOTVTCXNXSRBMT22VOA, seg_pos={0 0}, upload=478224]: Long tail cancellation due to optimal threshold Dec 28 08:50:12.926505349 (545h21m12.797588384s) [noise=true , address=45.91.116.5:4298 , piece_id=MAUITCSTYF5J6PHOWI2PICOYY36RAG4DH7OVP7TF3JXYPP3T2YJA, seg_pos={0 0}, upload=478224]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 28 08:50:12.926513163 (545h21m12.797596198s) [noise=true , address=45.91.116.5:4298 , piece_id=MAUITCSTYF5J6PHOWI2PICOYY36RAG4DH7OVP7TF3JXYPP3T2YJA, seg_pos={0 0}, upload=478224]: Long tail cancellation due to optimal threshold Dec 28 08:50:12.926555704 (545h21m12.797638749s) [seg_pos={0 0}, upload=478224]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 08:50:12.926696422 (545h21m12.797779477s) [upload=478224]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 08:50:12.926721089 (545h21m12.797804144s) [upload=478224]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 08:50:13.223494458 (545h21m13.094577503s) [upload=478224]: Done uploading segments... Dec 28 08:50:13.223516159 (545h21m13.094599205s) [upload=478224]: Upload finished. err: <nil> Duration: 5.942253481s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [X.XX.......X.XX.XX.......XX.X....XXXX.XX....X....X.XX..XX..X] duration: 5.000676674s trace: https://jaeger.dev.storj.io/trace/77e06878e3b86651 FAIL (2025-12-28 08:51:22 UTC) orbiter warning: context deadline exceeded Duration: 5.000700871s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 12.472945484s trace: https://jaeger.dev.storj.io/trace/177689f80539b862 FAIL (2025-12-28 08:51:21 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 12.472972725s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [XX.XX.......X.XX.XX.......XX.X....XXXX.XX....X....X.XX..XX..] duration: 4.879198258s trace: https://jaeger.dev.storj.io/trace/33170fda58ea5148 FAIL (2025-12-28 08:51:05 UTC) orbiter warning: context deadline exceeded Duration: 4.879223145s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [XX.XXX..X.XX.X.XX..X..XX....X...X.XX.XX.......X..X......XXXX] duration: 30.003861768s trace: https://jaeger.dev.storj.io/trace/667a4aac6d050408 FAIL (2025-12-28 08:50:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: YBXC3CEXJP3OTBYU3FMIMJSK4LPETZX55D3Z7VPR6NWY4YSEMBEA) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: OZOKGP6JN7CL6PJLWXDPG3QA2C5P5CMPMDDYOY2TVJMPKELNSBQQ) context deadline exceeded ======LOGS====== 08:50:10.285593840 (545h21m10.156676885s) [upload=478224]: Starting upload Dec 28 08:50:10.285688090 (545h21m10.156771155s) [upload=478224]: Done starting upload Dec 28 08:50:10.285836021 (545h21m10.156919056s) [upload=478224]: Uploading segments... Dec 28 08:50:10.285962142 (545h21m10.157045177s) [upload=478224]: Got next segment. Inline: false Dec 28 08:50:10.286117087 (545h21m10.157200132s) [upload=478224]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 08:50:10.286124731 (545h21m10.157207776s) [upload=478224]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 08:50:10.530460730 (545h21m10.401543775s) [seg_pos={0 0}, upload=478224]: Begin upload segment... Dec 28 08:50:10.530525243 (545h21m10.401608288s) [seg_pos={0 0}, upload=478224]: Done begin upload segment. Dec 28 08:50:10.530606627 (545h21m10.401689662s) [noise=true , address=77.90.31.19:4844 , piece_id=YBXC3CEXJP3OTBYU3FMIMJSK4LPETZX55D3Z7VPR6NWY4YSEMBEA, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.530634621 (545h21m10.401717666s) [noise=true , address=145.79.227.39:4013 , piece_id=JP34XIE6QZG2HJBONLUDACPYKCDTZQZW35LP2YC2VNBEBMDDWB3Q, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.531201670 (545h21m10.402284715s) [noise=true , address=145.79.227.37:4566 , piece_id=46FHYBKJ7KKE7XJSIYA7S5R23ZKB34UDUUV7PGTY7E2SJAS24SZQ, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.531776175 (545h21m10.40285922s) [noise=true , address=45.91.116.6:4975 , piece_id=QEREQPQV6F373QXQWDPQTBKW3BBOBZOB5FJ2XQWFY3BIPNV6JEDQ, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.532188299 (545h21m10.403271344s) [noise=true , address=45.91.116.8:4046 , piece_id=HVBKKQ4C75MYGYO42HFWI2RU35WC4RSX7BOTVTCXNXSRBMT22VOA, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.532624019 (545h21m10.403707054s) [noise=true , address=145.79.227.43:4159 , piece_id=IKFEA6LC4UNKJAR5EJTH5TNOTHAYXO34U5XWUTIP2MLLUDNDS6FA, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.533073034 (545h21m10.404156079s) [noise=true , address=145.79.227.48:4302 , piece_id=5C5N5KZV6QVPZFA6AZEBD2NOG7EJQ5XL437I6MKH2MOEJRGZCZ4Q, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.533540063 (545h21m10.404623108s) [noise=true , address=145.79.227.42:4121 , piece_id=WHDYTPU6FIBPH754UPZYQHV4AE5AH4S5X2UXU4IYLTOX62LSPSIA, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.535227467 (545h21m10.406310512s) [noise=true , address=145.79.227.17:4970 , piece_id=EBKOHG6YB3CKCAWAIBS6KW6NBTPIE2T5LOB4O3U2UX2V7GOVGCUA, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.536932173 (545h21m10.408015218s) [noise=true , address=145.79.227.11:4729 , piece_id=6KUUB7QM65FXIN2V6MNJJFC5B2PYV4MYUC4XB2CI6YMWU7CIWQ5Q, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.538421599 (545h21m10.409504644s) [noise=true , address=145.79.227.15:4873 , piece_id=ZZ2Y6VTARGFW4ROQOIA7SD34RWJRZGVTSN6NYYBWCMXIB4RBTBNQ, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.539724079 (545h21m10.410807124s) [noise=true , address=145.79.227.6:4259 , piece_id=CRXBD7264VZQLYJDJUPHKKJGFFWGDWYFUBW5V3PFENIR6KHRJIGA, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.540913543 (545h21m10.411996589s) [noise=true , address=145.79.227.13:4804 , piece_id=7RYCAZVU6BWZDBDJBSJ6CL57HY2EIY52RMII7QPC2LGWSR6YMXTQ, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.542259998 (545h21m10.413343033s) [noise=true , address=45.91.116.10:4084 , piece_id=OZOKGP6JN7CL6PJLWXDPG3QA2C5P5CMPMDDYOY2TVJMPKELNSBQQ, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.543536067 (545h21m10.414619102s) [noise=true , address=145.79.227.47:4265 , piece_id=JGPSKTCYBXM2WR2PQNYYSTGEESJIRWAMF5PFLH66XSU752XUF7RQ, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:10.547930976 (545h21m10.419014011s) [noise=true , address=45.91.116.5:4298 , piece_id=MAUITCSTYF5J6PHOWI2PICOYY36RAG4DH7OVP7TF3JXYPP3T2YJA, seg_pos={0 0}, upload=478224]: Uploading piece... Dec 28 08:50:12.223866421 (545h21m12.094949466s) [upload=478224]: Next returned nil segment Dec 28 08:50:12.223883273 (545h21m12.094966308s) [upload=478224]: Waiting for error group managing segments... Dec 28 08:50:12.539177488 (545h21m12.410260523s) [noise=true , address=145.79.227.13:4804 , piece_id=7RYCAZVU6BWZDBDJBSJ6CL57HY2EIY52RMII7QPC2LGWSR6YMXTQ, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.549298942 (545h21m12.420382007s) [noise=true , address=145.79.227.48:4302 , piece_id=5C5N5KZV6QVPZFA6AZEBD2NOG7EJQ5XL437I6MKH2MOEJRGZCZ4Q, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.553637723 (545h21m12.424720758s) [noise=true , address=145.79.227.39:4013 , piece_id=JP34XIE6QZG2HJBONLUDACPYKCDTZQZW35LP2YC2VNBEBMDDWB3Q, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.553678732 (545h21m12.424761857s) [noise=true , address=145.79.227.42:4121 , piece_id=WHDYTPU6FIBPH754UPZYQHV4AE5AH4S5X2UXU4IYLTOX62LSPSIA, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.554408450 (545h21m12.425491495s) [noise=true , address=145.79.227.6:4259 , piece_id=CRXBD7264VZQLYJDJUPHKKJGFFWGDWYFUBW5V3PFENIR6KHRJIGA, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.555001991 (545h21m12.426085026s) [noise=true , address=145.79.227.47:4265 , piece_id=JGPSKTCYBXM2WR2PQNYYSTGEESJIRWAMF5PFLH66XSU752XUF7RQ, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.555157477 (545h21m12.426240522s) [noise=true , address=145.79.227.17:4970 , piece_id=EBKOHG6YB3CKCAWAIBS6KW6NBTPIE2T5LOB4O3U2UX2V7GOVGCUA, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.567732742 (545h21m12.438815787s) [noise=true , address=145.79.227.15:4873 , piece_id=ZZ2Y6VTARGFW4ROQOIA7SD34RWJRZGVTSN6NYYBWCMXIB4RBTBNQ, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.569506931 (545h21m12.440589976s) [noise=true , address=145.79.227.37:4566 , piece_id=46FHYBKJ7KKE7XJSIYA7S5R23ZKB34UDUUV7PGTY7E2SJAS24SZQ, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.570004098 (545h21m12.441087143s) [noise=true , address=145.79.227.11:4729 , piece_id=6KUUB7QM65FXIN2V6MNJJFC5B2PYV4MYUC4XB2CI6YMWU7CIWQ5Q, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.805556120 (545h21m12.676639165s) [noise=true , address=77.90.31.19:4844 , piece_id=YBXC3CEXJP3OTBYU3FMIMJSK4LPETZX55D3Z7VPR6NWY4YSEMBEA, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.823831730 (545h21m12.694914775s) [noise=true , address=45.91.116.10:4084 , piece_id=OZOKGP6JN7CL6PJLWXDPG3QA2C5P5CMPMDDYOY2TVJMPKELNSBQQ, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.925559107 (545h21m12.796642152s) [noise=true , address=45.91.116.6:4975 , piece_id=QEREQPQV6F373QXQWDPQTBKW3BBOBZOB5FJ2XQWFY3BIPNV6JEDQ, seg_pos={0 0}, upload=478224]: Done uploading piece. err: <nil> Dec 28 08:50:12.925585126 (545h21m12.796668181s) [seg_pos={0 0}, upload=478224]: Segment reached optimal threshold of 13 pieces. Dec 28 08:50:12.925969327 (545h21m12.797052373s) [noise=true , address=145.79.227.43:4159 , piece_id=IKFEA6LC4UNKJAR5EJTH5TNOTHAYXO34U5XWUTIP2MLLUDNDS6FA, seg_pos={0 0}, upload=478224]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF Dec 28 08:50:12.925979888 (545h21m12.797062933s) [noise=true , address=145.79.227.43:4159 , piece_id=IKFEA6LC4UNKJAR5EJTH5TNOTHAYXO34U5XWUTIP2MLLUDNDS6FA, seg_pos={0 0}, upload=478224]: Long tail cancellation due to optimal threshold Dec 28 08:50:12.926202612 (545h21m12.797285667s) [noise=true , address=45.91.116.8:4046 , piece_id=HVBKKQ4C75MYGYO42HFWI2RU35WC4RSX7BOTVTCXNXSRBMT22VOA, seg_pos={0 0}, upload=478224]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: write tcp 188.166.225.170:46342->45.91.116.8:4046: use of closed network connection; write tcp 188.166.225.170:46342->45.91.116.8:4046: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:46342->45.91.116.8:4046: use of closed network connection Dec 28 08:50:12.926224965 (545h21m12.79730801s) [noise=true , address=45.91.116.8:4046 , piece_id=HVBKKQ4C75MYGYO42HFWI2RU35WC4RSX7BOTVTCXNXSRBMT22VOA, seg_pos={0 0}, upload=478224]: Long tail cancellation due to optimal threshold Dec 28 08:50:12.926505349 (545h21m12.797588384s) [noise=true , address=45.91.116.5:4298 , piece_id=MAUITCSTYF5J6PHOWI2PICOYY36RAG4DH7OVP7TF3JXYPP3T2YJA, seg_pos={0 0}, upload=478224]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 28 08:50:12.926513163 (545h21m12.797596198s) [noise=true , address=45.91.116.5:4298 , piece_id=MAUITCSTYF5J6PHOWI2PICOYY36RAG4DH7OVP7TF3JXYPP3T2YJA, seg_pos={0 0}, upload=478224]: Long tail cancellation due to optimal threshold Dec 28 08:50:12.926555704 (545h21m12.797638749s) [seg_pos={0 0}, upload=478224]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 08:50:12.926696422 (545h21m12.797779477s) [upload=478224]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 08:50:12.926721089 (545h21m12.797804144s) [upload=478224]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 08:50:13.223494458 (545h21m13.094577503s) [upload=478224]: Done uploading segments... Dec 28 08:50:13.223516159 (545h21m13.094599205s) [upload=478224]: Upload finished. err: <nil> Duration: 30.003873221s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 11.191808343s trace: https://jaeger.dev.storj.io/trace/177689f80539b862 FAIL (2025-12-28 08:50:18 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 11.191824133s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [.XX.XXX..X.XX.X.XX..X..XX....X...X.XX.XX.......X..X......XXX] duration: 30.003489959s trace: https://jaeger.dev.storj.io/trace/51d481331aa7c6f0 FAIL (2025-12-28 08:50:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 08:49:40.073775366 (545h20m39.944858411s) [upload=478216]: Starting upload Dec 28 08:49:40.073885557 (545h20m39.944968592s) [upload=478216]: Done starting upload Dec 28 08:49:40.073939699 (545h20m39.945022734s) [upload=478216]: Uploading segments... Dec 28 08:49:40.074254339 (545h20m39.945337374s) [upload=478216]: Got next segment. Inline: false Dec 28 08:49:40.074292471 (545h20m39.945375506s) [upload=478216]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 08:49:40.074298392 (545h20m39.945381427s) [upload=478216]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 08:49:40.319967249 (545h20m40.191050284s) [seg_pos={0 0}, upload=478216]: Begin upload segment... Dec 28 08:49:40.320010591 (545h20m40.191093636s) [seg_pos={0 0}, upload=478216]: Done begin upload segment. Dec 28 08:49:40.320034888 (545h20m40.191117923s) [noise=true , address=145.79.227.36:4517 , piece_id=6CCCHWJE6TKCESAK6QNO42UBOTPMWQSZO2DS5CKHJNLHTH4KU3ZQ, seg_pos={0 0}, upload=478216]: Uploading piece... Dec 28 08:49:40.320286687 (545h20m40.191369722s) [noise=true , address=145.79.227.17:4972 , piece_id=XQQCCE5JNG6AJRM5W7DU44QTWGO3LRDC22UH2ZB53LTB425VEQZQ, seg_pos={0 0}, upload=478216]: Uploading piece... Dec 28 08:49:40.322329998 (545h20m40.193413043s) [noise=true , address=145.79.227.47:4266 , piece_id=WT7B3LI7LQXX3JS2UXHUT5RF6QX2Y6DYCBIBQW3RDWSE55L4PN6Q, seg_pos={0 0}, upload=478216]: Uploading piece... Dec 28 08:49:40.322796766 (545h20m40.193879811s) [noise=true , address=145.79.227.13:4803 , piece_id=X33RTBIFX7RHVSUQQ77CIGJGYNIVM2EEOYZMH7ILXY26P7OHQUTA, seg_pos={0 0}, upload=478216]: Uploading piece... Dec 28 08:49:40.323949312 (545h20m40.195032347s) [noise=true , address=145.79.227.11:4729 , piece_id=FKIA7N44DQM2AENAMHXNL36YJP5MVH4XTCGDUNGIV5CSJZU33QQA, seg_pos={0 0}, upload=478216]: Uploading piece... Dec 28 08:49:40.324584181 (545h20m40.195667236s) [noise=true , address=77.90.31.11:4555 , piece_id=NWFEP6EBC4QMZHUQKTKFJFRVGID4OI2JYULTA4NYWSNM7JJTRAAQ, seg_pos={0 0}, upload=478216]: Uploading piece... Dec 28 08:49:40.325328477 (545h20m40.196411513s) [noise=true , address=145.79.227.23:4547 , piece_id=AZ5YVRCI67L5IMIQZU3GSCSPJL3FZZQDKAPFZAC2JUBQJFHB5IKA, seg_pos={0 0}, upload=478216]: Uploading piece... Dec 28 08:49:40.327147230 (545h20m40.198230266s) [noise=true , address=45.91.116.4:4939 , piece_id=UQUAR2X33JKOQ3YW3ABKOVUN46CLBICKYKNPW2WQWFHBFDSFM4DQ, seg_pos={0 0}, upload=478216]: Uploading piece... Dec 28 08:49:40.327618939 (545h20m40.198701984s) [noise=true , address=45.91.116.6:4975 , piece_id=3RXNEJ3JGLQ75IQTD3EDZH3WLVO7HUYIP7DIRT3URASKHLYIV63Q, seg_pos={0 0}, upload=478216]: Uploading piece... Dec 28 08:49:40.330369406 (545h20m40.201452452s) [noise=true , address=145.79.227.41:4085 , piece_id=RNJ7H6MLQJ3Z45HFDE7H6SKO5YVTO6RJL5IJJQT2NSTY7DYE4EFA, seg_pos={0 0}, upload=478216]: Uploading piece... Dec 28 08:49:40.331043900 (545h20m40.202126946s) [noise=true , address=145.79.227.37:4567 , piece_id=5PW4JF6KXY4PZYF6QRSYHV5DZ2IH2SSPO27HEVUZSVAATIQ6G2SA, seg_pos={0 0}, upload=478216]: Uploading piece... Dec 28 08:49:40.333950616 (545h20m40.205033651s) [noise=true , address=45.91.116.9:4261 , piece_id=OQ5U5RMD7ZGQU2JLEE5QK47YKH6VKYDFUVB56L6SMSMTWAM73MPQ, seg_pos={0 0}, upload=478216]: Uploading piece... Dec 28 08:49:40.334377428 (545h20m40.205460463s) [noise=true , address=77.90.31.17:4773 , piece_id=FTRMBZDYX2HT5N7ZGTJX26ZBMORNO2PV6TVZLNRVYJGFT56244JA, seg_pos={0 0}, upload=478216]: Uploading piece... Dec 28 08:49:40.335278885 (545h20m40.20636193s) [noise=true , address=77.90.31.18:4805 , piece_id=IV4IQ74STLQUKIHBSQLNYX6SHHN7C532TJX4ZCY6RHTGUGHNWT6Q, seg_pos={0 0}, upload=478216]: Uploading piece... Dec 28 08:49:40.335965022 (545h20m40.207048057s) [noise=true , address=77.90.31.16:4736 , piece_id=ONXFWCI6ORRON4ATQK7PEZ5OAE3AA2RNR2BEQMGH3W5NK5CZJIJQ, seg_pos={0 0}, upload=478216]: Uploading piece... Dec 28 08:49:40.338754473 (545h20m40.209837518s) [noise=true , address=145.79.227.31:4797 , piece_id=DYQ65NOOT7ALL4RXAJSNMOEDE22SNUA2LY4CJQBSQS3XITW2XOFA, seg_pos={0 0}, upload=478216]: Uploading piece... Dec 28 08:49:42.055239913 (545h20m41.926322978s) [upload=478216]: Next returned nil segment Dec 28 08:49:42.055266553 (545h20m41.926349598s) [upload=478216]: Waiting for error group managing segments... Dec 28 08:49:42.341771749 (545h20m42.212854784s) [noise=true , address=145.79.227.36:4517 , piece_id=6CCCHWJE6TKCESAK6QNO42UBOTPMWQSZO2DS5CKHJNLHTH4KU3ZQ, seg_pos={0 0}, upload=478216]: Done uploading piece. err: <nil> Dec 28 08:49:42.343358000 (545h20m42.214441045s) [noise=true , address=145.79.227.17:4972 , piece_id=XQQCCE5JNG6AJRM5W7DU44QTWGO3LRDC22UH2ZB53LTB425VEQZQ, seg_pos={0 0}, upload=478216]: Done uploading piece. err: <nil> Dec 28 08:49:42.343460375 (545h20m42.21454341s) [noise=true , address=145.79.227.11:4729 , piece_id=FKIA7N44DQM2AENAMHXNL36YJP5MVH4XTCGDUNGIV5CSJZU33QQA, seg_pos={0 0}, upload=478216]: Done uploading piece. err: <nil> Dec 28 08:49:42.347097249 (545h20m42.218180284s) [noise=true , address=145.79.227.31:4797 , piece_id=DYQ65NOOT7ALL4RXAJSNMOEDE22SNUA2LY4CJQBSQS3XITW2XOFA, seg_pos={0 0}, upload=478216]: Done uploading piece. err: <nil> Dec 28 08:49:42.348307174 (545h20m42.219390209s) [noise=true , address=145.79.227.23:4547 , piece_id=AZ5YVRCI67L5IMIQZU3GSCSPJL3FZZQDKAPFZAC2JUBQJFHB5IKA, seg_pos={0 0}, upload=478216]: Done uploading piece. err: <nil> Dec 28 08:49:42.358414088 (545h20m42.229497124s) [noise=true , address=145.79.227.47:4266 , piece_id=WT7B3LI7LQXX3JS2UXHUT5RF6QX2Y6DYCBIBQW3RDWSE55L4PN6Q, seg_pos={0 0}, upload=478216]: Done uploading piece. err: <nil> Dec 28 08:49:42.359988778 (545h20m42.231071813s) [noise=true , address=145.79.227.41:4085 , piece_id=RNJ7H6MLQJ3Z45HFDE7H6SKO5YVTO6RJL5IJJQT2NSTY7DYE4EFA, seg_pos={0 0}, upload=478216]: Done uploading piece. err: <nil> Dec 28 08:49:42.362108564 (545h20m42.233191599s) [noise=true , address=145.79.227.37:4567 , piece_id=5PW4JF6KXY4PZYF6QRSYHV5DZ2IH2SSPO27HEVUZSVAATIQ6G2SA, seg_pos={0 0}, upload=478216]: Done uploading piece. err: <nil> Dec 28 08:49:42.366883647 (545h20m42.237966682s) [noise=true , address=145.79.227.13:4803 , piece_id=X33RTBIFX7RHVSUQQ77CIGJGYNIVM2EEOYZMH7ILXY26P7OHQUTA, seg_pos={0 0}, upload=478216]: Done uploading piece. err: <nil> Dec 28 08:49:42.535233213 (545h20m42.406316248s) [noise=true , address=77.90.31.17:4773 , piece_id=FTRMBZDYX2HT5N7ZGTJX26ZBMORNO2PV6TVZLNRVYJGFT56244JA, seg_pos={0 0}, upload=478216]: Done uploading piece. err: <nil> Dec 28 08:49:42.581178872 (545h20m42.452261907s) [noise=true , address=45.91.116.9:4261 , piece_id=OQ5U5RMD7ZGQU2JLEE5QK47YKH6VKYDFUVB56L6SMSMTWAM73MPQ, seg_pos={0 0}, upload=478216]: Done uploading piece. err: <nil> Dec 28 08:49:42.620939603 (545h20m42.492022638s) [noise=true , address=77.90.31.18:4805 , piece_id=IV4IQ74STLQUKIHBSQLNYX6SHHN7C532TJX4ZCY6RHTGUGHNWT6Q, seg_pos={0 0}, upload=478216]: Done uploading piece. err: <nil> Dec 28 08:49:42.648989375 (545h20m42.52007241s) [noise=true , address=77.90.31.16:4736 , piece_id=ONXFWCI6ORRON4ATQK7PEZ5OAE3AA2RNR2BEQMGH3W5NK5CZJIJQ, seg_pos={0 0}, upload=478216]: Done uploading piece. err: <nil> Dec 28 08:49:42.649014242 (545h20m42.520097297s) [seg_pos={0 0}, upload=478216]: Segment reached optimal threshold of 13 pieces. Dec 28 08:49:42.649319444 (545h20m42.520402479s) [noise=true , address=77.90.31.11:4555 , piece_id=NWFEP6EBC4QMZHUQKTKFJFRVGID4OI2JYULTA4NYWSNM7JJTRAAQ, seg_pos={0 0}, upload=478216]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ): protocol: expected piece hash; context canceled; EOF Dec 28 08:49:42.649333891 (545h20m42.520416936s) [noise=true , address=77.90.31.11:4555 , piece_id=NWFEP6EBC4QMZHUQKTKFJFRVGID4OI2JYULTA4NYWSNM7JJTRAAQ, seg_pos={0 0}, upload=478216]: Long tail cancellation due to optimal threshold Dec 28 08:49:42.649347967 (545h20m42.520431002s) [noise=true , address=45.91.116.6:4975 , piece_id=3RXNEJ3JGLQ75IQTD3EDZH3WLVO7HUYIP7DIRT3URASKHLYIV63Q, seg_pos={0 0}, upload=478216]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF Dec 28 08:49:42.649354661 (545h20m42.520437696s) [noise=true , address=45.91.116.6:4975 , piece_id=3RXNEJ3JGLQ75IQTD3EDZH3WLVO7HUYIP7DIRT3URASKHLYIV63Q, seg_pos={0 0}, upload=478216]: Long tail cancellation due to optimal threshold Dec 28 08:49:42.649573076 (545h20m42.520656111s) [noise=true , address=45.91.116.4:4939 , piece_id=UQUAR2X33JKOQ3YW3ABKOVUN46CLBICKYKNPW2WQWFHBFDSFM4DQ, seg_pos={0 0}, upload=478216]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Dec 28 08:49:42.649581452 (545h20m42.520664487s) [noise=true , address=45.91.116.4:4939 , piece_id=UQUAR2X33JKOQ3YW3ABKOVUN46CLBICKYKNPW2WQWFHBFDSFM4DQ, seg_pos={0 0}, upload=478216]: Long tail cancellation due to optimal threshold Dec 28 08:49:42.649600739 (545h20m42.520683774s) [seg_pos={0 0}, upload=478216]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 08:49:42.649707833 (545h20m42.520790868s) [upload=478216]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 08:49:42.649715878 (545h20m42.520798923s) [upload=478216]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 08:49:42.958085848 (545h20m42.829168903s) [upload=478216]: Done uploading segments... Dec 28 08:49:42.958111117 (545h20m42.829194162s) [upload=478216]: Upload finished. err: <nil> Duration: 30.003515558s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [.XX.XX.......X.XX.XX.......XX.X....XXXX.XX....X....X.XX..XX.] duration: 5.00154623s trace: https://jaeger.dev.storj.io/trace/33170fda58ea5148 FAIL (2025-12-28 08:49:57 UTC) orbiter warning: context deadline exceeded Duration: 5.001571438s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 12.319622509s trace: https://jaeger.dev.storj.io/trace/177689f80539b862 FAIL (2025-12-28 08:49:46 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 12.319646865s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [X.XX.XXX..X.XX.X.XX..X..XX....X...X.XX.XX.......X..X......XX] duration: 30.002261613s trace: https://jaeger.dev.storj.io/trace/4500399b6ba75cc0 FAIL (2025-12-28 08:49:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: MOW36H52SFE7H74NEATSSG3OZIMSIKV5JUYJFXC2MU63GT7K4O3Q) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: WV4QXZB6BP3S4SBIY6KPXTMRL23FDMQ226E44P5UEDKJGQJZMHBQ) context deadline exceeded ======LOGS====== ts... Dec 28 08:48:56.076695419 (545h19m55.947778454s) [upload=478207]: Got next segment. Inline: false Dec 28 08:48:56.076737829 (545h19m55.947820864s) [upload=478207]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 08:48:56.076742197 (545h19m55.947825232s) [upload=478207]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 08:48:56.319365936 (545h19m56.190448981s) [seg_pos={0 0}, upload=478207]: Begin upload segment... Dec 28 08:48:56.319452420 (545h19m56.190535455s) [seg_pos={0 0}, upload=478207]: Done begin upload segment. Dec 28 08:48:56.319509950 (545h19m56.190593015s) [noise=true , address=45.91.116.11:4118 , piece_id=6DR4KW2ZVG5B7B2BWBD4C7FCUMGV62ZBQFDUTFBDZC3K6DMSEACQ, seg_pos={0 0}, upload=478207]: Uploading piece... Dec 28 08:48:56.319542472 (545h19m56.190625507s) [noise=true , address=145.79.227.19:4078 , piece_id=BLM6ZE6UVKQ2LDQVDHIGQNTUHTERKCDNMMO2O56AS6IPWAJRONGQ, seg_pos={0 0}, upload=478207]: Uploading piece... Dec 28 08:48:56.319622023 (545h19m56.190705058s) [noise=true , address=145.79.227.16:4911 , piece_id=LHHJFU4GCR25YTPLDKHLSC33MJLED2GIKCLKDGHPURTY6KSPBQMQ, seg_pos={0 0}, upload=478207]: Uploading piece... Dec 28 08:48:56.320069866 (545h19m56.191152911s) [noise=true , address=145.79.227.29:4943 , piece_id=4NYPYQA4P4VCOP2RVUD54P7CUSDDWYRYFQX2C3YBD6SSTOGH5YAQ, seg_pos={0 0}, upload=478207]: Uploading piece... Dec 28 08:48:56.320439900 (545h19m56.191522935s) [noise=true , address=145.79.227.28:4727 , piece_id=77M5WZCES4ASTGH7QYLW6QUHEOI7ND2ZMN6N7W3GQIYUX5VYHUUA, seg_pos={0 0}, upload=478207]: Uploading piece... Dec 28 08:48:56.320530653 (545h19m56.191613699s) [noise=true , address=45.91.116.6:4973 , piece_id=MOW36H52SFE7H74NEATSSG3OZIMSIKV5JUYJFXC2MU63GT7K4O3Q, seg_pos={0 0}, upload=478207]: Uploading piece... Dec 28 08:48:56.320900518 (545h19m56.191983563s) [noise=true , address=45.91.116.13:4153 , piece_id=6T2ZEAJL54ANNLAUQIPBTADNVLLERICHUKMZOJ6GOO7GDDQKOJKA, seg_pos={0 0}, upload=478207]: Uploading piece... Dec 28 08:48:56.321002973 (545h19m56.192086008s) [noise=true , address=145.79.227.48:4301 , piece_id=AHBQ4RGVAUAIOXYRJI65L6IJGQJLPXLOSMTNK6NFPEH2S7QQBEFQ, seg_pos={0 0}, upload=478207]: Uploading piece... Dec 28 08:48:56.321280210 (545h19m56.192363245s) [noise=true , address=145.79.227.20:4437 , piece_id=URDEASFPBRK6ASDX2FXNYJECIAD3X7OJVFIU2X5AGCRJZ3C62QFQ, seg_pos={0 0}, upload=478207]: Uploading piece... Dec 28 08:48:56.321446357 (545h19m56.192529482s) [noise=true , address=145.79.227.18:4007 , piece_id=D2PZ2WPDP7ISC5PYQURY7U3VQBYZRNYUTME5N4VPOBCFEAH2JHWQ, seg_pos={0 0}, upload=478207]: Uploading piece... Dec 28 08:48:56.321625708 (545h19m56.192708753s) [noise=true , address=145.79.227.6:4259 , piece_id=Z4TI2PFAVT4S2CN4LVK6RMZQJFWKRAWMT7IAMD4Y4RHLLXZWGL5Q, seg_pos={0 0}, upload=478207]: Uploading piece... Dec 28 08:48:56.321964043 (545h19m56.193047078s) [noise=true , address=45.91.116.12:4227 , piece_id=WV4QXZB6BP3S4SBIY6KPXTMRL23FDMQ226E44P5UEDKJGQJZMHBQ, seg_pos={0 0}, upload=478207]: Uploading piece... Dec 28 08:48:56.322250097 (545h19m56.193333132s) [noise=true , address=45.91.116.7:4009 , piece_id=H536RANP6UJ2VRM4VM2F7L7JO7YSK6GLOJQ6H6NNIDAQWWHWHBIQ, seg_pos={0 0}, upload=478207]: Uploading piece... Dec 28 08:48:56.322555369 (545h19m56.193638404s) [noise=true , address=145.79.227.43:4158 , piece_id=M6LM4CKVJAQG66ALX3TCQRRQLV3NDJPKQYKM3DHU4OGWHC2OMX4A, seg_pos={0 0}, upload=478207]: Uploading piece... Dec 28 08:48:56.322604383 (545h19m56.193687428s) [noise=true , address=77.90.31.19:4842 , piece_id=PTPQQIZTXXWKEWB5VOK42LTRVPTYLRX3X4H6FNBH2KDR5UGMIKBQ, seg_pos={0 0}, upload=478207]: Uploading piece... Dec 28 08:48:56.323019302 (545h19m56.194102347s) [noise=true , address=145.79.227.52:4446 , piece_id=W5ZPAIBRYV2RZWTZMMRT6Y6IZBSJCSLVQZV2PSG7ITBRP4XDPUMQ, seg_pos={0 0}, upload=478207]: Uploading piece... Dec 28 08:48:58.043187264 (545h19m57.914270299s) [upload=478207]: Next returned nil segment Dec 28 08:48:58.043200308 (545h19m57.914283343s) [upload=478207]: Waiting for error group managing segments... Dec 28 08:48:58.330153638 (545h19m58.201236693s) [noise=true , address=145.79.227.18:4007 , piece_id=D2PZ2WPDP7ISC5PYQURY7U3VQBYZRNYUTME5N4VPOBCFEAH2JHWQ, seg_pos={0 0}, upload=478207]: Done uploading piece. err: <nil> Dec 28 08:48:58.335366073 (545h19m58.206449118s) [noise=true , address=145.79.227.6:4259 , piece_id=Z4TI2PFAVT4S2CN4LVK6RMZQJFWKRAWMT7IAMD4Y4RHLLXZWGL5Q, seg_pos={0 0}, upload=478207]: Done uploading piece. err: <nil> Dec 28 08:48:58.341966391 (545h19m58.213049436s) [noise=true , address=145.79.227.16:4911 , piece_id=LHHJFU4GCR25YTPLDKHLSC33MJLED2GIKCLKDGHPURTY6KSPBQMQ, seg_pos={0 0}, upload=478207]: Done uploading piece. err: <nil> Dec 28 08:48:58.345180492 (545h19m58.216263537s) [noise=true , address=145.79.227.48:4301 , piece_id=AHBQ4RGVAUAIOXYRJI65L6IJGQJLPXLOSMTNK6NFPEH2S7QQBEFQ, seg_pos={0 0}, upload=478207]: Done uploading piece. err: <nil> Dec 28 08:48:58.345600141 (545h19m58.216683176s) [noise=true , address=145.79.227.52:4446 , piece_id=W5ZPAIBRYV2RZWTZMMRT6Y6IZBSJCSLVQZV2PSG7ITBRP4XDPUMQ, seg_pos={0 0}, upload=478207]: Done uploading piece. err: <nil> Dec 28 08:48:58.348478322 (545h19m58.219561367s) [noise=true , address=145.79.227.20:4437 , piece_id=URDEASFPBRK6ASDX2FXNYJECIAD3X7OJVFIU2X5AGCRJZ3C62QFQ, seg_pos={0 0}, upload=478207]: Done uploading piece. err: <nil> Dec 28 08:48:58.348630531 (545h19m58.219713566s) [noise=true , address=145.79.227.19:4078 , piece_id=BLM6ZE6UVKQ2LDQVDHIGQNTUHTERKCDNMMO2O56AS6IPWAJRONGQ, seg_pos={0 0}, upload=478207]: Done uploading piece. err: <nil> Dec 28 08:48:58.349244601 (545h19m58.220327646s) [noise=true , address=145.79.227.43:4158 , piece_id=M6LM4CKVJAQG66ALX3TCQRRQLV3NDJPKQYKM3DHU4OGWHC2OMX4A, seg_pos={0 0}, upload=478207]: Done uploading piece. err: <nil> Dec 28 08:48:58.352065722 (545h19m58.223148758s) [noise=true , address=145.79.227.28:4727 , piece_id=77M5WZCES4ASTGH7QYLW6QUHEOI7ND2ZMN6N7W3GQIYUX5VYHUUA, seg_pos={0 0}, upload=478207]: Done uploading piece. err: <nil> Dec 28 08:48:58.353874677 (545h19m58.224966639s) [noise=true , address=145.79.227.29:4943 , piece_id=4NYPYQA4P4VCOP2RVUD54P7CUSDDWYRYFQX2C3YBD6SSTOGH5YAQ, seg_pos={0 0}, upload=478207]: Done uploading piece. err: <nil> Dec 28 08:48:58.580651319 (545h19m58.451734354s) [noise=true , address=45.91.116.12:4227 , piece_id=WV4QXZB6BP3S4SBIY6KPXTMRL23FDMQ226E44P5UEDKJGQJZMHBQ, seg_pos={0 0}, upload=478207]: Done uploading piece. err: <nil> Dec 28 08:48:58.611981738 (545h19m58.483064773s) [noise=true , address=45.91.116.13:4153 , piece_id=6T2ZEAJL54ANNLAUQIPBTADNVLLERICHUKMZOJ6GOO7GDDQKOJKA, seg_pos={0 0}, upload=478207]: Done uploading piece. err: <nil> Dec 28 08:48:58.639705208 (545h19m58.510788243s) [noise=true , address=45.91.116.6:4973 , piece_id=MOW36H52SFE7H74NEATSSG3OZIMSIKV5JUYJFXC2MU63GT7K4O3Q, seg_pos={0 0}, upload=478207]: Done uploading piece. err: <nil> Dec 28 08:48:58.639725316 (545h19m58.510808352s) [seg_pos={0 0}, upload=478207]: Segment reached optimal threshold of 13 pieces. Dec 28 08:48:58.639968770 (545h19m58.511051815s) [noise=true , address=45.91.116.7:4009 , piece_id=H536RANP6UJ2VRM4VM2F7L7JO7YSK6GLOJQ6H6NNIDAQWWHWHBIQ, seg_pos={0 0}, upload=478207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: write tcp 188.166.225.170:33036->45.91.116.7:4009: use of closed network connection; write tcp 188.166.225.170:33036->45.91.116.7:4009: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33036->45.91.116.7:4009: use of closed network connection Dec 28 08:48:58.639978669 (545h19m58.511061714s) [noise=true , address=45.91.116.7:4009 , piece_id=H536RANP6UJ2VRM4VM2F7L7JO7YSK6GLOJQ6H6NNIDAQWWHWHBIQ, seg_pos={0 0}, upload=478207]: Long tail cancellation due to optimal threshold Dec 28 08:48:58.640045606 (545h19m58.511128651s) [noise=true , address=77.90.31.19:4842 , piece_id=PTPQQIZTXXWKEWB5VOK42LTRVPTYLRX3X4H6FNBH2KDR5UGMIKBQ, seg_pos={0 0}, upload=478207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 28 08:48:58.640053341 (545h19m58.511136376s) [noise=true , address=77.90.31.19:4842 , piece_id=PTPQQIZTXXWKEWB5VOK42LTRVPTYLRX3X4H6FNBH2KDR5UGMIKBQ, seg_pos={0 0}, upload=478207]: Long tail cancellation due to optimal threshold Dec 28 08:48:58.640204288 (545h19m58.511287333s) [noise=true , address=45.91.116.11:4118 , piece_id=6DR4KW2ZVG5B7B2BWBD4C7FCUMGV62ZBQFDUTFBDZC3K6DMSEACQ, seg_pos={0 0}, upload=478207]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: write tcp 188.166.225.170:53734->45.91.116.11:4118: use of closed network connection; write tcp 188.166.225.170:53734->45.91.116.11:4118: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53734->45.91.116.11:4118: use of closed network connection Dec 28 08:48:58.640209989 (545h19m58.511293025s) [noise=true , address=45.91.116.11:4118 , piece_id=6DR4KW2ZVG5B7B2BWBD4C7FCUMGV62ZBQFDUTFBDZC3K6DMSEACQ, seg_pos={0 0}, upload=478207]: Long tail cancellation due to optimal threshold Dec 28 08:48:58.640226731 (545h19m58.511309766s) [seg_pos={0 0}, upload=478207]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 08:48:58.640313406 (545h19m58.511396441s) [upload=478207]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 08:48:58.640318506 (545h19m58.511401551s) [upload=478207]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 08:48:58.924974160 (545h19m58.796057205s) [upload=478207]: Done uploading segments... Dec 28 08:48:58.924994499 (545h19m58.796077534s) [upload=478207]: Upload finished. err: <nil> Duration: 30.002279937s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 25 out of 60 is failed (threshold=6) [X.XX.XX.......X.XX.XX.......XX.X....XXXX.XX....X....X.XX..XX] duration: 5.001040066s trace: https://jaeger.dev.storj.io/trace/33170fda58ea5148 FAIL (2025-12-28 08:49:18 UTC) orbiter warning: context deadline exceeded Duration: 5.001064022s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [.X.XX.XX.......X.XX.XX.......XX.X....XXXX.XX....X....X.XX..X] duration: 5.001402267s trace: https://jaeger.dev.storj.io/trace/3fc347e9faa4016b FAIL (2025-12-28 08:48:54 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00145131s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 14.747986433s trace: https://jaeger.dev.storj.io/trace/177689f80539b862 FAIL (2025-12-28 08:48:42 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 14.748012043s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [XX.XX.XXX..X.XX.X.XX..X..XX....X...X.XX.XX.......X..X......X] duration: 30.002812531s trace: https://jaeger.dev.storj.io/trace/66f42cdc6770b8ef FAIL (2025-12-28 08:48:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2, Piece ID: 2XPMYPVJLLH3NSLWXWYOPFG7LBXZR4OZJ4J5UY3OUJTP7KKQU7AA) context deadline exceeded; piecestore: (Node ID: 1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, Piece ID: LK3KJYIBOF3NQSOUUE5MUGP2RMF2BV5DKP3RHM73FOFQDTHUHGFQ) context deadline exceeded ======LOGS====== egments... Dec 28 08:47:52.063792667 (545h18m51.934875702s) [upload=478194]: Got next segment. Inline: false Dec 28 08:47:52.063833334 (545h18m51.934916369s) [upload=478194]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 08:47:52.063840098 (545h18m51.934923143s) [upload=478194]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 08:47:52.313574055 (545h18m52.184657101s) [seg_pos={0 0}, upload=478194]: Begin upload segment... Dec 28 08:47:52.313630473 (545h18m52.184713508s) [seg_pos={0 0}, upload=478194]: Done begin upload segment. Dec 28 08:47:52.313672423 (545h18m52.184755458s) [noise=true , address=77.90.31.21:4877 , piece_id=2XPMYPVJLLH3NSLWXWYOPFG7LBXZR4OZJ4J5UY3OUJTP7KKQU7AA, seg_pos={0 0}, upload=478194]: Uploading piece... Dec 28 08:47:52.313721686 (545h18m52.184804721s) [noise=true , address=45.91.116.9:4262 , piece_id=WS2IUJPA4IMIV5NJYRUEXH4Y5IK7FTNUOHFDSBGYEHTZJPCWGQPQ, seg_pos={0 0}, upload=478194]: Uploading piece... Dec 28 08:47:52.314638282 (545h18m52.185721317s) [noise=true , address=45.91.116.13:4154 , piece_id=JKWSMNAL6R4R3SSWJT3UKTXU5EXSCS4JCBNFSYJQD4VLZRHTKVEQ, seg_pos={0 0}, upload=478194]: Uploading piece... Dec 28 08:47:52.315140027 (545h18m52.186223072s) [noise=true , address=45.91.116.4:4940 , piece_id=2AXHV4RHT5Y3L7PJQLSIIJK5VVA7DDHFUIX6L7UXYA6SX65CSY6Q, seg_pos={0 0}, upload=478194]: Uploading piece... Dec 28 08:47:52.315710013 (545h18m52.186793068s) [noise=true , address=145.79.227.39:4014 , piece_id=BFBGWHGIQZVYT75ICMMOOSZK7AAXW5PXY23XILHNVYZPIKKZSRGA, seg_pos={0 0}, upload=478194]: Uploading piece... Dec 28 08:47:52.316089074 (545h18m52.187172109s) [noise=true , address=45.91.116.8:4045 , piece_id=LK3KJYIBOF3NQSOUUE5MUGP2RMF2BV5DKP3RHM73FOFQDTHUHGFQ, seg_pos={0 0}, upload=478194]: Uploading piece... Dec 28 08:47:52.316415527 (545h18m52.187498562s) [noise=true , address=145.79.227.3:4151 , piece_id=4XCMC4CBFO7YL5A6TUOECULYXMMVTONIHK5GY6OMDXL5QSWCPVOA, seg_pos={0 0}, upload=478194]: Uploading piece... Dec 28 08:47:52.317378019 (545h18m52.188461064s) [noise=true , address=145.79.227.16:4911 , piece_id=CHO4H4XSODSGRXALNVPSOOLZGNL53BO252PLLDJFYS6PXN3ECP6Q, seg_pos={0 0}, upload=478194]: Uploading piece... Dec 28 08:47:52.317445647 (545h18m52.188528682s) [noise=true , address=145.79.227.13:4804 , piece_id=MNEYK6DDOBUZY2L5TPPQNDBJPHZD4BIJ3LFPVM7LUAV3NDMXHRQQ, seg_pos={0 0}, upload=478194]: Uploading piece... Dec 28 08:47:52.318715446 (545h18m52.189798481s) [noise=true , address=77.90.31.14:4662 , piece_id=37LJTVDH5UUJEP7LQP32AL6YAMZGTHTVPXXZS62FGN5AO47JLMIQ, seg_pos={0 0}, upload=478194]: Uploading piece... Dec 28 08:47:52.319072305 (545h18m52.19015535s) [noise=true , address=145.79.227.24:4581 , piece_id=WUXCCRKUEVQSNE6DEXGZNWZDW7J2DCU3I32SZCOEUZAAKQVJPMBA, seg_pos={0 0}, upload=478194]: Uploading piece... Dec 28 08:47:52.319856469 (545h18m52.190939514s) [noise=true , address=145.79.227.45:4229 , piece_id=NIDKUPNSJJASZSLDNL3KEL25LKFZMGZ6SRBEGEHRSPDORIQZPIZQ, seg_pos={0 0}, upload=478194]: Uploading piece... Dec 28 08:47:52.320967615 (545h18m52.19205065s) [noise=true , address=145.79.227.51:4409 , piece_id=ZUNFEEURPH2RYL3473ZW3YZBNQVWQWVMLX2BHV56GPUVMPUKNH3A, seg_pos={0 0}, upload=478194]: Uploading piece... Dec 28 08:47:52.322106493 (545h18m52.193189538s) [noise=true , address=145.79.227.32:4835 , piece_id=D5SKKFDJK6LRRZYDKYO3522DLHUO2W46I56WGMRWSJKJAM2UE75A, seg_pos={0 0}, upload=478194]: Uploading piece... Dec 28 08:47:52.322308988 (545h18m52.193392023s) [noise=true , address=45.91.116.10:4084 , piece_id=DW2WQJDXWX6KMUVTQY7WI47ISX4FZFLOTSM6BONZOSKMSIWZKYXQ, seg_pos={0 0}, upload=478194]: Uploading piece... Dec 28 08:47:52.323652988 (545h18m52.194736023s) [noise=true , address=145.79.227.52:4446 , piece_id=XU57KDV42NORTFPVDPOESDBI3YHURJ2Q7BJBCJB2CAM46YMIXHBA, seg_pos={0 0}, upload=478194]: Uploading piece... Dec 28 08:47:54.040846144 (545h18m53.911929189s) [upload=478194]: Next returned nil segment Dec 28 08:47:54.040860701 (545h18m53.911943736s) [upload=478194]: Waiting for error group managing segments... Dec 28 08:47:54.328796372 (545h18m54.199879407s) [noise=true , address=145.79.227.24:4581 , piece_id=WUXCCRKUEVQSNE6DEXGZNWZDW7J2DCU3I32SZCOEUZAAKQVJPMBA, seg_pos={0 0}, upload=478194]: Done uploading piece. err: <nil> Dec 28 08:47:54.330914785 (545h18m54.20199783s) [noise=true , address=145.79.227.51:4409 , piece_id=ZUNFEEURPH2RYL3473ZW3YZBNQVWQWVMLX2BHV56GPUVMPUKNH3A, seg_pos={0 0}, upload=478194]: Done uploading piece. err: <nil> Dec 28 08:47:54.332123046 (545h18m54.203206091s) [noise=true , address=145.79.227.3:4151 , piece_id=4XCMC4CBFO7YL5A6TUOECULYXMMVTONIHK5GY6OMDXL5QSWCPVOA, seg_pos={0 0}, upload=478194]: Done uploading piece. err: <nil> Dec 28 08:47:54.332388812 (545h18m54.203471857s) [noise=true , address=145.79.227.32:4835 , piece_id=D5SKKFDJK6LRRZYDKYO3522DLHUO2W46I56WGMRWSJKJAM2UE75A, seg_pos={0 0}, upload=478194]: Done uploading piece. err: <nil> Dec 28 08:47:54.333567006 (545h18m54.204650051s) [noise=true , address=145.79.227.52:4446 , piece_id=XU57KDV42NORTFPVDPOESDBI3YHURJ2Q7BJBCJB2CAM46YMIXHBA, seg_pos={0 0}, upload=478194]: Done uploading piece. err: <nil> Dec 28 08:47:54.344655600 (545h18m54.215738635s) [noise=true , address=145.79.227.13:4804 , piece_id=MNEYK6DDOBUZY2L5TPPQNDBJPHZD4BIJ3LFPVM7LUAV3NDMXHRQQ, seg_pos={0 0}, upload=478194]: Done uploading piece. err: <nil> Dec 28 08:47:54.348928156 (545h18m54.220011191s) [noise=true , address=145.79.227.45:4229 , piece_id=NIDKUPNSJJASZSLDNL3KEL25LKFZMGZ6SRBEGEHRSPDORIQZPIZQ, seg_pos={0 0}, upload=478194]: Done uploading piece. err: <nil> Dec 28 08:47:54.349693022 (545h18m54.220776057s) [noise=true , address=145.79.227.16:4911 , piece_id=CHO4H4XSODSGRXALNVPSOOLZGNL53BO252PLLDJFYS6PXN3ECP6Q, seg_pos={0 0}, upload=478194]: Done uploading piece. err: <nil> Dec 28 08:47:54.353959938 (545h18m54.225042983s) [noise=true , address=145.79.227.39:4014 , piece_id=BFBGWHGIQZVYT75ICMMOOSZK7AAXW5PXY23XILHNVYZPIKKZSRGA, seg_pos={0 0}, upload=478194]: Done uploading piece. err: <nil> Dec 28 08:47:54.555256932 (545h18m54.426339967s) [noise=true , address=45.91.116.8:4045 , piece_id=LK3KJYIBOF3NQSOUUE5MUGP2RMF2BV5DKP3RHM73FOFQDTHUHGFQ, seg_pos={0 0}, upload=478194]: Done uploading piece. err: <nil> Dec 28 08:47:54.588019538 (545h18m54.459102573s) [noise=true , address=77.90.31.21:4877 , piece_id=2XPMYPVJLLH3NSLWXWYOPFG7LBXZR4OZJ4J5UY3OUJTP7KKQU7AA, seg_pos={0 0}, upload=478194]: Done uploading piece. err: <nil> Dec 28 08:47:54.606324514 (545h18m54.477407559s) [noise=true , address=77.90.31.14:4662 , piece_id=37LJTVDH5UUJEP7LQP32AL6YAMZGTHTVPXXZS62FGN5AO47JLMIQ, seg_pos={0 0}, upload=478194]: Done uploading piece. err: <nil> Dec 28 08:47:54.640116339 (545h18m54.511199384s) [noise=true , address=45.91.116.10:4084 , piece_id=DW2WQJDXWX6KMUVTQY7WI47ISX4FZFLOTSM6BONZOSKMSIWZKYXQ, seg_pos={0 0}, upload=478194]: Done uploading piece. err: <nil> Dec 28 08:47:54.640142198 (545h18m54.511225263s) [seg_pos={0 0}, upload=478194]: Segment reached optimal threshold of 13 pieces. Dec 28 08:47:54.640374831 (545h18m54.511457876s) [noise=true , address=45.91.116.13:4154 , piece_id=JKWSMNAL6R4R3SSWJT3UKTXU5EXSCS4JCBNFSYJQD4VLZRHTKVEQ, seg_pos={0 0}, upload=478194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 28 08:47:54.640384229 (545h18m54.511467264s) [noise=true , address=45.91.116.13:4154 , piece_id=JKWSMNAL6R4R3SSWJT3UKTXU5EXSCS4JCBNFSYJQD4VLZRHTKVEQ, seg_pos={0 0}, upload=478194]: Long tail cancellation due to optimal threshold Dec 28 08:47:54.640470273 (545h18m54.511553318s) [noise=true , address=45.91.116.4:4940 , piece_id=2AXHV4RHT5Y3L7PJQLSIIJK5VVA7DDHFUIX6L7UXYA6SX65CSY6Q, seg_pos={0 0}, upload=478194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:44950->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:44950->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44950->45.91.116.4:4940: use of closed network connection Dec 28 08:47:54.640482536 (545h18m54.511565591s) [noise=true , address=45.91.116.4:4940 , piece_id=2AXHV4RHT5Y3L7PJQLSIIJK5VVA7DDHFUIX6L7UXYA6SX65CSY6Q, seg_pos={0 0}, upload=478194]: Long tail cancellation due to optimal threshold Dec 28 08:47:54.640650096 (545h18m54.511733141s) [noise=true , address=45.91.116.9:4262 , piece_id=WS2IUJPA4IMIV5NJYRUEXH4Y5IK7FTNUOHFDSBGYEHTZJPCWGQPQ, seg_pos={0 0}, upload=478194]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: write tcp 188.166.225.170:53980->45.91.116.9:4262: use of closed network connection; write tcp 188.166.225.170:53980->45.91.116.9:4262: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53980->45.91.116.9:4262: use of closed network connection Dec 28 08:47:54.640660836 (545h18m54.511743881s) [noise=true , address=45.91.116.9:4262 , piece_id=WS2IUJPA4IMIV5NJYRUEXH4Y5IK7FTNUOHFDSBGYEHTZJPCWGQPQ, seg_pos={0 0}, upload=478194]: Long tail cancellation due to optimal threshold Dec 28 08:47:54.640681465 (545h18m54.511764511s) [seg_pos={0 0}, upload=478194]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 08:47:54.640794771 (545h18m54.511877816s) [upload=478194]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 08:47:54.640803247 (545h18m54.511886312s) [upload=478194]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 08:47:54.939536318 (545h18m54.810619353s) [upload=478194]: Done uploading segments... Dec 28 08:47:54.939553510 (545h18m54.810636545s) [upload=478194]: Upload finished. err: <nil> Duration: 30.00283842s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 24 out of 60 is failed (threshold=6) [X.X.XX.XX.......X.XX.XX.......XX.X....XXXX.XX....X....X.XX..] duration: 4.263114496s trace: https://jaeger.dev.storj.io/trace/154549febed4f317 FAIL (2025-12-28 08:47:41 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.263177075s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 10.816043883s trace: https://jaeger.dev.storj.io/trace/177689f80539b862 FAIL (2025-12-28 08:47:37 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout Duration: 10.816074932s edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)