Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [XX..X...X.XX.X...X.........X.X........XX.X...X..X....XXX...X] duration:
30.002675478s trace:
https://jaeger.dev.storj.io/trace/63f9c1d68d003623
FAIL (2026-01-01 08:09:28 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Jan 1 08:08:59.168503451 (640h39m59.039586486s) [upload=561368]: Starting upload
Jan 1 08:08:59.168576611 (640h39m59.039659646s) [upload=561368]: Done starting upload
Jan 1 08:08:59.168667714 (640h39m59.03975075s) [upload=561368]: Uploading segments...
Jan 1 08:08:59.168781000 (640h39m59.039864045s) [upload=561368]: Got next segment. Inline: false
Jan 1 08:08:59.168815315 (640h39m59.03989835s) [upload=561368]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 1 08:08:59.168820976 (640h39m59.039904011s) [upload=561368]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 1 08:08:59.417407948 (640h39m59.288490983s) [seg_pos={0 0}, upload=561368]: Begin upload segment...
Jan 1 08:08:59.417488812 (640h39m59.288571857s) [seg_pos={0 0}, upload=561368]: Done begin upload segment.
Jan 1 08:08:59.417516004 (640h39m59.288599039s) [noise=true , address=145.79.227.6:4258 , piece_id=R5UJ5HH5CGJCFRL5XLHTYBEP65JSIKJTDMXEXOWBWGKS5CQRDQXQ, seg_pos={0 0}, upload=561368]: Uploading piece...
Jan 1 08:08:59.417783372 (640h39m59.288866417s) [noise=true , address=77.90.31.19:4844 , piece_id=FETSATLYOM3AFJYPUWCXHITJP5Z6UMNEYLD6U7LRODZJVMBQXVFA, seg_pos={0 0}, upload=561368]: Uploading piece...
Jan 1 08:08:59.418810549 (640h39m59.289893584s) [noise=true , address=45.91.116.10:4084 , piece_id=XDL7YXALCNQYMT7TXY76ORNB3JZQEI5PV525FWPLV53VAVEF422A, seg_pos={0 0}, upload=561368]: Uploading piece...
Jan 1 08:08:59.419209217 (640h39m59.290292263s) [noise=true , address=145.79.227.10:4403 , piece_id=OPJFQUOAM6M663GFSPH2T7FPPRPAKQK6RSVMH3C7PDFKYPPDMPAQ, seg_pos={0 0}, upload=561368]: Uploading piece...
Jan 1 08:08:59.419264794 (640h39m59.290347839s) [noise=true , address=145.79.227.9:4366 , piece_id=5ETY5FPYQJR7MSAOBCTWOC5WQH3P6UODUB4CXXBEWCJT3BYTFPHA, seg_pos={0 0}, upload=561368]: Uploading piece...
Jan 1 08:08:59.419348984 (640h39m59.290432029s) [noise=true , address=145.79.227.50:4375 , piece_id=4TVRIXU63GGGKW5FO2PB35JO4LCIMNT5WGZDDWHO4NEJCMKR3TNQ, seg_pos={0 0}, upload=561368]: Uploading piece...
Jan 1 08:08:59.419906104 (640h39m59.29098914s) [noise=true , address=145.79.227.26:4655 , piece_id=T5GAMG46T55NWD46HK3FIFJ2W4S24MZBQZWJBVZKI3CIIL7KIQXQ, seg_pos={0 0}, upload=561368]: Uploading piece...
Jan 1 08:08:59.420695117 (640h39m59.291778152s) [noise=true , address=145.79.227.18:4007 , piece_id=H2FJQCPMYY75DFBNCOP7IBRWX2NXEQ5UUSPVXIFR3YZAFR6ROU5A, seg_pos={0 0}, upload=561368]: Uploading piece...
Jan 1 08:08:59.421488848 (640h39m59.292571893s) [noise=true , address=45.91.116.11:4119 , piece_id=6NYLOIYWUCF4IHKO5T5FW2IWJ4YC6G3RR6N5GE7GBWSW5XBJ5GWA, seg_pos={0 0}, upload=561368]: Uploading piece...
Jan 1 08:08:59.422287509 (640h39m59.293370554s) [noise=true , address=145.79.227.22:4511 , piece_id=T3RSTOKOF34M3UJ7DJEPDQPLBDR4SYM5BJCQEIEKIGYWKWMNH4YA, seg_pos={0 0}, upload=561368]: Uploading piece...
Jan 1 08:08:59.425263626 (640h39m59.296346661s) [noise=true , address=145.79.227.3:4149 , piece_id=3LERMEOAL3JICXV3SDSDUJJXVP6CQGXOYY66QL63JKPORMKJ4NKQ, seg_pos={0 0}, upload=561368]: Uploading piece...
Jan 1 08:08:59.438035275 (640h39m59.30911831s) [noise=true , address=77.90.31.21:4877 , piece_id=QA6BFB725SSZPZ2FVGWNNVYZAODQHCBB3OEDI5D7KWGKRMSFWUBQ, seg_pos={0 0}, upload=561368]: Uploading piece...
Jan 1 08:08:59.441119417 (640h39m59.312202452s) [noise=true , address=145.79.227.45:4230 , piece_id=JAUFMFBDIEJU44GCT3REF3ORRJXM6DU62CFQGGTSEA63ZUC5UIHA, seg_pos={0 0}, upload=561368]: Uploading piece...
Jan 1 08:08:59.441986197 (640h39m59.313069233s) [noise=true , address=77.90.31.12:4589 , piece_id=ZEPU3CLH5VF26M5XZC5XXKPPT4AKUXSRATNR5O6FADSCVZOQG4QQ, seg_pos={0 0}, upload=561368]: Uploading piece...
Jan 1 08:08:59.443161135 (640h39m59.31424417s) [noise=true , address=145.79.227.41:4086 , piece_id=37JLN4TJYNUQ4D6CHT5MV6VOA7LOGMGYZJEHZTUU7AWPTG3G7PVQ, seg_pos={0 0}, upload=561368]: Uploading piece...
Jan 1 08:08:59.445214755 (640h39m59.316297791s) [noise=true , address=45.91.116.12:4226 , piece_id=OATBYUNJLPPJHVWCJASCSJQ7VZOI2RUYYHSR6DXVWSWC6TMSV3FQ, seg_pos={0 0}, upload=561368]: Uploading piece...
Jan 1 08:09:01.136754195 (640h40m1.00783724s) [upload=561368]: Next returned nil segment
Jan 1 08:09:01.136773192 (640h40m1.007856237s) [upload=561368]: Waiting for error group managing segments...
Jan 1 08:09:01.415067719 (640h40m1.286150764s) [noise=true , address=145.79.227.6:4258 , piece_id=R5UJ5HH5CGJCFRL5XLHTYBEP65JSIKJTDMXEXOWBWGKS5CQRDQXQ, seg_pos={0 0}, upload=561368]: Done uploading piece. err: <nil>
Jan 1 08:09:01.425666230 (640h40m1.296749275s) [noise=true , address=145.79.227.26:4655 , piece_id=T5GAMG46T55NWD46HK3FIFJ2W4S24MZBQZWJBVZKI3CIIL7KIQXQ, seg_pos={0 0}, upload=561368]: Done uploading piece. err: <nil>
Jan 1 08:09:01.428312299 (640h40m1.299395344s) [noise=true , address=145.79.227.18:4007 , piece_id=H2FJQCPMYY75DFBNCOP7IBRWX2NXEQ5UUSPVXIFR3YZAFR6ROU5A, seg_pos={0 0}, upload=561368]: Done uploading piece. err: <nil>
Jan 1 08:09:01.430830043 (640h40m1.301913088s) [noise=true , address=145.79.227.9:4366 , piece_id=5ETY5FPYQJR7MSAOBCTWOC5WQH3P6UODUB4CXXBEWCJT3BYTFPHA, seg_pos={0 0}, upload=561368]: Done uploading piece. err: <nil>
Jan 1 08:09:01.432711896 (640h40m1.303794941s) [noise=true , address=145.79.227.50:4375 , piece_id=4TVRIXU63GGGKW5FO2PB35JO4LCIMNT5WGZDDWHO4NEJCMKR3TNQ, seg_pos={0 0}, upload=561368]: Done uploading piece. err: <nil>
Jan 1 08:09:01.438372334 (640h40m1.309455379s) [noise=true , address=145.79.227.10:4403 , piece_id=OPJFQUOAM6M663GFSPH2T7FPPRPAKQK6RSVMH3C7PDFKYPPDMPAQ, seg_pos={0 0}, upload=561368]: Done uploading piece. err: <nil>
Jan 1 08:09:01.440580930 (640h40m1.311663975s) [noise=true , address=145.79.227.45:4230 , piece_id=JAUFMFBDIEJU44GCT3REF3ORRJXM6DU62CFQGGTSEA63ZUC5UIHA, seg_pos={0 0}, upload=561368]: Done uploading piece. err: <nil>
Jan 1 08:09:01.441613255 (640h40m1.3126963s) [noise=true , address=145.79.227.41:4086 , piece_id=37JLN4TJYNUQ4D6CHT5MV6VOA7LOGMGYZJEHZTUU7AWPTG3G7PVQ, seg_pos={0 0}, upload=561368]: Done uploading piece. err: <nil>
Jan 1 08:09:01.535909377 (640h40m1.406992432s) [noise=true , address=145.79.227.3:4149 , piece_id=3LERMEOAL3JICXV3SDSDUJJXVP6CQGXOYY66QL63JKPORMKJ4NKQ, seg_pos={0 0}, upload=561368]: Done uploading piece. err: <nil>
Jan 1 08:09:01.584322451 (640h40m1.455405496s) [noise=true , address=145.79.227.22:4511 , piece_id=T3RSTOKOF34M3UJ7DJEPDQPLBDR4SYM5BJCQEIEKIGYWKWMNH4YA, seg_pos={0 0}, upload=561368]: Done uploading piece. err: <nil>
Jan 1 08:09:01.646839632 (640h40m1.517922677s) [noise=true , address=77.90.31.12:4589 , piece_id=ZEPU3CLH5VF26M5XZC5XXKPPT4AKUXSRATNR5O6FADSCVZOQG4QQ, seg_pos={0 0}, upload=561368]: Done uploading piece. err: <nil>
Jan 1 08:09:01.729529985 (640h40m1.60061304s) [noise=true , address=77.90.31.21:4877 , piece_id=QA6BFB725SSZPZ2FVGWNNVYZAODQHCBB3OEDI5D7KWGKRMSFWUBQ, seg_pos={0 0}, upload=561368]: Done uploading piece. err: <nil>
Jan 1 08:09:01.765114222 (640h40m1.636197267s) [noise=true , address=45.91.116.10:4084 , piece_id=XDL7YXALCNQYMT7TXY76ORNB3JZQEI5PV525FWPLV53VAVEF422A, seg_pos={0 0}, upload=561368]: Done uploading piece. err: <nil>
Jan 1 08:09:01.765145892 (640h40m1.636228937s) [seg_pos={0 0}, upload=561368]: Segment reached optimal threshold of 13 pieces.
Jan 1 08:09:01.765472463 (640h40m1.636555508s) [noise=true , address=77.90.31.19:4844 , piece_id=FETSATLYOM3AFJYPUWCXHITJP5Z6UMNEYLD6U7LRODZJVMBQXVFA, seg_pos={0 0}, upload=561368]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t): protocol: expected piece hash; context canceled; EOF
Jan 1 08:09:01.765484327 (640h40m1.636567382s) [noise=true , address=77.90.31.19:4844 , piece_id=FETSATLYOM3AFJYPUWCXHITJP5Z6UMNEYLD6U7LRODZJVMBQXVFA, seg_pos={0 0}, upload=561368]: Long tail cancellation due to optimal threshold
Jan 1 08:09:01.765536996 (640h40m1.636620052s) [noise=true , address=45.91.116.11:4119 , piece_id=6NYLOIYWUCF4IHKO5T5FW2IWJ4YC6G3RR6N5GE7GBWSW5XBJ5GWA, seg_pos={0 0}, upload=561368]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: expected piece hash; context canceled; EOF
Jan 1 08:09:01.765550101 (640h40m1.636633156s) [noise=true , address=45.91.116.11:4119 , piece_id=6NYLOIYWUCF4IHKO5T5FW2IWJ4YC6G3RR6N5GE7GBWSW5XBJ5GWA, seg_pos={0 0}, upload=561368]: Long tail cancellation due to optimal threshold
Jan 1 08:09:01.765880841 (640h40m1.636963896s) [noise=true , address=45.91.116.12:4226 , piece_id=OATBYUNJLPPJHVWCJASCSJQ7VZOI2RUYYHSR6DXVWSWC6TMSV3FQ, seg_pos={0 0}, upload=561368]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF
Jan 1 08:09:01.765893625 (640h40m1.63697667s) [noise=true , address=45.91.116.12:4226 , piece_id=OATBYUNJLPPJHVWCJASCSJQ7VZOI2RUYYHSR6DXVWSWC6TMSV3FQ, seg_pos={0 0}, upload=561368]: Long tail cancellation due to optimal threshold
Jan 1 08:09:01.766082065 (640h40m1.63716512s) [seg_pos={0 0}, upload=561368]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 1 08:09:01.766286023 (640h40m1.637369078s) [upload=561368]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 1 08:09:01.766294028 (640h40m1.637377073s) [upload=561368]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 1 08:09:02.073237538 (640h40m1.944320593s) [upload=561368]: Done uploading segments...
Jan 1 08:09:02.073264839 (640h40m1.944347884s) [upload=561368]: Upload finished. err: <nil>
Duration: 30.002696818s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
7 out of 60 is failed (threshold=3) [.........X.....X..XX.........................XX..........X..] duration:
28.921254755s trace:
https://jaeger.dev.storj.io/trace/e95ca73698c6b71
FAIL (2026-01-01 08:09:21 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
.227.43:4182 , piece_id=GXY3PEKWQSHZ2WJHJEF54XU75BD2MHL5XYYRXCTVOKD3BFOFY33A, seg_pos={0 0}, upload=561335]: Uploading piece...
Jan 1 08:06:39.265949369 (640h37m39.137032404s) [noise=true , address=64.49.13.170:4718 , piece_id=MFZMTGAUBC65VN6KKLZDUCXI7RF6QZWGOCOHWJMEBITCXIL6Q5QQ, seg_pos={0 0}, upload=561335]: Uploading piece...
Jan 1 08:06:39.266002000 (640h37m39.137085045s) [noise=true , address=77.90.31.13:4654 , piece_id=JC34MCI3U7Y7Y5JDRYHRU6ACPYQNNJKMDIK3FME5VWMUAUBRFVDA, seg_pos={0 0}, upload=561335]: Uploading piece...
Jan 1 08:06:39.266075890 (640h37m39.137158925s) [noise=true , address=145.79.227.35:4498 , piece_id=MZ6FBJE3D2ZVNO54464FUQ6AN2RCR36Z7YVPOWNB3X25UJFZ7OGQ, seg_pos={0 0}, upload=561335]: Uploading piece...
Jan 1 08:06:39.266104886 (640h37m39.137187931s) [noise=true , address=145.79.227.40:4067 , piece_id=RNGN65VKKQZ3A7OPI2VKA5EBSYCKXMHVQGO6YJXG2YRX5EQEYCQA, seg_pos={0 0}, upload=561335]: Uploading piece...
Jan 1 08:06:39.266729564 (640h37m39.137812599s) [noise=true , address=77.90.31.12:4610 , piece_id=KRA6U6RAZLXQNOBW6HZL7QLLQK7GJXADHOC3QQH55QIE5T2SI7KA, seg_pos={0 0}, upload=561335]: Uploading piece...
Jan 1 08:06:39.266817723 (640h37m39.137900758s) [noise=true , address=145.79.227.30:4776 , piece_id=BLBTAEJLWTL4O2FCB27WERV4FN666F6XT6HXQOD563TYUCVJOLOQ, seg_pos={0 0}, upload=561335]: Uploading piece...
Jan 1 08:06:39.267202215 (640h37m39.13828526s) [noise=true , address=145.79.227.29:4973 , piece_id=SOSOH3ZHT4OFVIWXS27EPVJFJCI64VTD44HPNISPYKSBBD5ALNAA, seg_pos={0 0}, upload=561335]: Uploading piece...
Jan 1 08:06:39.267270585 (640h37m39.13835363s) [noise=true , address=45.91.116.9:4266 , piece_id=3RUSCCKAC36NILU7FVTBVDUO72PLGNU3RUSZJOHLYEQDI2QD6RBA, seg_pos={0 0}, upload=561335]: Uploading piece...
Jan 1 08:06:39.267548815 (640h37m39.13863185s) [noise=true , address=145.79.227.44:4213 , piece_id=ENIA5SMSCD7QX6P4ELVIJAFTJIYSZ4RDP7KPKWQ6COKUZF3WXJDQ, seg_pos={0 0}, upload=561335]: Uploading piece...
Jan 1 08:06:39.267565676 (640h37m39.138648721s) [noise=true , address=45.91.116.6:4001 , piece_id=IA43L7PEUXG2F64REP4GZWURKMEJBWZBCYQJXMGDPLZX2FJBYYZQ, seg_pos={0 0}, upload=561335]: Uploading piece...
Jan 1 08:06:39.267812327 (640h37m39.138895372s) [noise=true , address=77.90.31.15:4732 , piece_id=NQ4YN73ZNXTQ3XSTIREDNOFNI4CDX6X4ZLINL2I65BPESTSXNK6A, seg_pos={0 0}, upload=561335]: Uploading piece...
Jan 1 08:06:39.267997900 (640h37m39.139080935s) [noise=true , address=193.149.164.143:4694 , piece_id=JWHH6WDJT2D4SX4Y2DUPJCXVYFHB5ITUANBG63YPDI5KCVFVASDA, seg_pos={0 0}, upload=561335]: Uploading piece...
Jan 1 08:06:39.268112398 (640h37m39.139195443s) [noise=true , address=145.79.227.49:4369 , piece_id=BSVLOVXXRKQROP56TPYDPTV6YRG6WGCTHKER3ZE76VQ3OUQLCG6Q, seg_pos={0 0}, upload=561335]: Uploading piece...
Jan 1 08:06:39.268406639 (640h37m39.139489684s) [noise=true , address=145.79.227.33:4880 , piece_id=WN2T2IFGZPE5ZWVCIU4AVXP4KUIFYWFZ56O5WA3EDVM2UPGDOJ2A, seg_pos={0 0}, upload=561335]: Uploading piece...
Jan 1 08:06:40.656816578 (640h37m40.527899623s) [upload=561335]: Next returned nil segment
Jan 1 08:06:40.656833560 (640h37m40.527916595s) [upload=561335]: Waiting for error group managing segments...
Jan 1 08:06:41.014218117 (640h37m40.885301162s) [noise=true , address=64.49.13.170:4718 , piece_id=MFZMTGAUBC65VN6KKLZDUCXI7RF6QZWGOCOHWJMEBITCXIL6Q5QQ, seg_pos={0 0}, upload=561335]: Done uploading piece. err: <nil>
Jan 1 08:06:41.157461380 (640h37m41.028544425s) [noise=true , address=145.79.227.13:4812 , piece_id=NQXXT7UJSYSOTQOU2TEX3M3RRQS7CJJN5KTUGEKFEXKSH2VY4Q7A, seg_pos={0 0}, upload=561335]: Done uploading piece. err: <nil>
Jan 1 08:06:41.331693451 (640h37m41.202776486s) [noise=true , address=77.90.31.15:4732 , piece_id=NQ4YN73ZNXTQ3XSTIREDNOFNI4CDX6X4ZLINL2I65BPESTSXNK6A, seg_pos={0 0}, upload=561335]: Done uploading piece. err: <nil>
Jan 1 08:06:41.332724885 (640h37m41.20380793s) [noise=true , address=145.79.227.35:4498 , piece_id=MZ6FBJE3D2ZVNO54464FUQ6AN2RCR36Z7YVPOWNB3X25UJFZ7OGQ, seg_pos={0 0}, upload=561335]: Done uploading piece. err: <nil>
Jan 1 08:06:41.382546381 (640h37m41.253629416s) [noise=true , address=77.90.31.12:4610 , piece_id=KRA6U6RAZLXQNOBW6HZL7QLLQK7GJXADHOC3QQH55QIE5T2SI7KA, seg_pos={0 0}, upload=561335]: Done uploading piece. err: <nil>
Jan 1 08:06:41.388091100 (640h37m41.259174135s) [noise=true , address=64.49.13.155:4914 , piece_id=R253V6KYITKWIYUNDNVS3EGVNIR5GKDBYAX67TWLFY67JZPJNKHQ, seg_pos={0 0}, upload=561335]: Done uploading piece. err: <nil>
Jan 1 08:06:41.563625079 (640h37m41.434708114s) [noise=true , address=45.91.116.9:4266 , piece_id=3RUSCCKAC36NILU7FVTBVDUO72PLGNU3RUSZJOHLYEQDI2QD6RBA, seg_pos={0 0}, upload=561335]: Done uploading piece. err: <nil>
Jan 1 08:06:41.911410376 (640h37m41.782493411s) [noise=true , address=193.149.164.143:4694 , piece_id=JWHH6WDJT2D4SX4Y2DUPJCXVYFHB5ITUANBG63YPDI5KCVFVASDA, seg_pos={0 0}, upload=561335]: Done uploading piece. err: <nil>
Jan 1 08:06:42.051037685 (640h37m41.92212073s) [noise=true , address=145.79.227.29:4973 , piece_id=SOSOH3ZHT4OFVIWXS27EPVJFJCI64VTD44HPNISPYKSBBD5ALNAA, seg_pos={0 0}, upload=561335]: Done uploading piece. err: <nil>
Jan 1 08:06:42.084648975 (640h37m41.95573201s) [noise=true , address=145.79.227.44:4213 , piece_id=ENIA5SMSCD7QX6P4ELVIJAFTJIYSZ4RDP7KPKWQ6COKUZF3WXJDQ, seg_pos={0 0}, upload=561335]: Done uploading piece. err: <nil>
Jan 1 08:06:42.096110790 (640h37m41.967193825s) [noise=true , address=145.79.227.49:4369 , piece_id=BSVLOVXXRKQROP56TPYDPTV6YRG6WGCTHKER3ZE76VQ3OUQLCG6Q, seg_pos={0 0}, upload=561335]: Done uploading piece. err: <nil>
Jan 1 08:06:42.104023627 (640h37m41.975106672s) [noise=true , address=145.79.227.30:4776 , piece_id=BLBTAEJLWTL4O2FCB27WERV4FN666F6XT6HXQOD563TYUCVJOLOQ, seg_pos={0 0}, upload=561335]: Done uploading piece. err: <nil>
Jan 1 08:06:42.109017707 (640h37m41.980100762s) [noise=true , address=145.79.227.40:4067 , piece_id=RNGN65VKKQZ3A7OPI2VKA5EBSYCKXMHVQGO6YJXG2YRX5EQEYCQA, seg_pos={0 0}, upload=561335]: Done uploading piece. err: <nil>
Jan 1 08:06:42.150694945 (640h37m42.02177798s) [noise=true , address=145.79.227.43:4182 , piece_id=GXY3PEKWQSHZ2WJHJEF54XU75BD2MHL5XYYRXCTVOKD3BFOFY33A, seg_pos={0 0}, upload=561335]: Done uploading piece. err: <nil>
Jan 1 08:06:42.150713521 (640h37m42.021796556s) [seg_pos={0 0}, upload=561335]: Segment reached optimal threshold of 14 pieces.
Jan 1 08:06:42.150857585 (640h37m42.021940631s) [noise=true , address=64.49.13.169:4716 , piece_id=YZGERUD3ZPPHAGU6MHCIDBSXELDFPFNEQ4A2CKBCZD2TTF7B5O6Q, seg_pos={0 0}, upload=561335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:47710->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:47710->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47710->64.49.13.169:4716: use of closed network connection
Jan 1 08:06:42.150862715 (640h37m42.02194575s) [noise=true , address=64.49.13.169:4716 , piece_id=YZGERUD3ZPPHAGU6MHCIDBSXELDFPFNEQ4A2CKBCZD2TTF7B5O6Q, seg_pos={0 0}, upload=561335]: Long tail cancellation due to optimal threshold
Jan 1 08:06:42.150906138 (640h37m42.021989173s) [noise=true , address=145.79.227.33:4880 , piece_id=WN2T2IFGZPE5ZWVCIU4AVXP4KUIFYWFZ56O5WA3EDVM2UPGDOJ2A, seg_pos={0 0}, upload=561335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sFbwzYfjUGQqrKJhgwDg5dvormM1Xu4yc97awR5kgZUAsKiEn): protocol: expected piece hash; context canceled; EOF
Jan 1 08:06:42.150915375 (640h37m42.02199841s) [noise=true , address=145.79.227.33:4880 , piece_id=WN2T2IFGZPE5ZWVCIU4AVXP4KUIFYWFZ56O5WA3EDVM2UPGDOJ2A, seg_pos={0 0}, upload=561335]: Long tail cancellation due to optimal threshold
Jan 1 08:06:42.151035294 (640h37m42.022118329s) [noise=true , address=45.91.116.6:4001 , piece_id=IA43L7PEUXG2F64REP4GZWURKMEJBWZBCYQJXMGDPLZX2FJBYYZQ, seg_pos={0 0}, upload=561335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQJmv9n2yWXZtCCaABrxkB3NN6vDwLdiBkRibDN3t6YNmcmGP): protocol: expected piece hash; context canceled; EOF
Jan 1 08:06:42.151046475 (640h37m42.02212951s) [noise=true , address=45.91.116.6:4001 , piece_id=IA43L7PEUXG2F64REP4GZWURKMEJBWZBCYQJXMGDPLZX2FJBYYZQ, seg_pos={0 0}, upload=561335]: Long tail cancellation due to optimal threshold
Jan 1 08:06:42.151055602 (640h37m42.022138637s) [noise=true , address=77.90.31.13:4654 , piece_id=JC34MCI3U7Y7Y5JDRYHRU6ACPYQNNJKMDIK3FME5VWMUAUBRFVDA, seg_pos={0 0}, upload=561335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8): protocol: write tcp 188.166.225.170:50358->77.90.31.13:4654: use of closed network connection; write tcp 188.166.225.170:50358->77.90.31.13:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50358->77.90.31.13:4654: use of closed network connection
Jan 1 08:06:42.151061083 (640h37m42.022144118s) [noise=true , address=77.90.31.13:4654 , piece_id=JC34MCI3U7Y7Y5JDRYHRU6ACPYQNNJKMDIK3FME5VWMUAUBRFVDA, seg_pos={0 0}, upload=561335]: Long tail cancellation due to optimal threshold
Jan 1 08:06:42.151075680 (640h37m42.022158716s) [seg_pos={0 0}, upload=561335]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 1 08:06:42.151139893 (640h37m42.022222928s) [upload=561335]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 1 08:06:42.151144381 (640h37m42.022227416s) [upload=561335]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 1 08:06:42.440313737 (640h37m42.311396782s) [upload=561335]: Done uploading segments...
Jan 1 08:06:42.440334847 (640h37m42.311417882s) [upload=561335]: Upload finished. err: <nil>
Duration: 28.921313407s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
11.812300938s trace:
https://jaeger.dev.storj.io/trace/44e9d74e3b154e8e
FAIL (2026-01-01 08:08:52 UTC)
orbiter warning: context deadline exceeded
Duration: 11.812326435s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
39 out of 60 is failed (threshold=6) [.XXXXX.XX..XXXXX.XXXX..XX.XXX.XX...X.X..X...XXXXXX..XXXXXXX.] duration:
5.001180093s trace:
https://jaeger.dev.storj.io/trace/18db0c1415df8ad9
FAIL (2026-01-01 08:08:32 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001197375s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
7 out of 60 is failed (threshold=3) [..........X.....X..XX.........................XX..........X.] duration:
33.156229841s trace:
https://jaeger.dev.storj.io/trace/e95ca73698c6b71
FAIL (2026-01-01 08:08:11 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
.227.43:4182 , piece_id=GXY3PEKWQSHZ2WJHJEF54XU75BD2MHL5XYYRXCTVOKD3BFOFY33A, seg_pos={0 0}, upload=561335]: Uploading piece...
Jan 1 08:06:39.265949369 (640h37m39.137032404s) [noise=true , address=64.49.13.170:4718 , piece_id=MFZMTGAUBC65VN6KKLZDUCXI7RF6QZWGOCOHWJMEBITCXIL6Q5QQ, seg_pos={0 0}, upload=561335]: Uploading piece...
Jan 1 08:06:39.266002000 (640h37m39.137085045s) [noise=true , address=77.90.31.13:4654 , piece_id=JC34MCI3U7Y7Y5JDRYHRU6ACPYQNNJKMDIK3FME5VWMUAUBRFVDA, seg_pos={0 0}, upload=561335]: Uploading piece...
Jan 1 08:06:39.266075890 (640h37m39.137158925s) [noise=true , address=145.79.227.35:4498 , piece_id=MZ6FBJE3D2ZVNO54464FUQ6AN2RCR36Z7YVPOWNB3X25UJFZ7OGQ, seg_pos={0 0}, upload=561335]: Uploading piece...
Jan 1 08:06:39.266104886 (640h37m39.137187931s) [noise=true , address=145.79.227.40:4067 , piece_id=RNGN65VKKQZ3A7OPI2VKA5EBSYCKXMHVQGO6YJXG2YRX5EQEYCQA, seg_pos={0 0}, upload=561335]: Uploading piece...
Jan 1 08:06:39.266729564 (640h37m39.137812599s) [noise=true , address=77.90.31.12:4610 , piece_id=KRA6U6RAZLXQNOBW6HZL7QLLQK7GJXADHOC3QQH55QIE5T2SI7KA, seg_pos={0 0}, upload=561335]: Uploading piece...
Jan 1 08:06:39.266817723 (640h37m39.137900758s) [noise=true , address=145.79.227.30:4776 , piece_id=BLBTAEJLWTL4O2FCB27WERV4FN666F6XT6HXQOD563TYUCVJOLOQ, seg_pos={0 0}, upload=561335]: Uploading piece...
Jan 1 08:06:39.267202215 (640h37m39.13828526s) [noise=true , address=145.79.227.29:4973 , piece_id=SOSOH3ZHT4OFVIWXS27EPVJFJCI64VTD44HPNISPYKSBBD5ALNAA, seg_pos={0 0}, upload=561335]: Uploading piece...
Jan 1 08:06:39.267270585 (640h37m39.13835363s) [noise=true , address=45.91.116.9:4266 , piece_id=3RUSCCKAC36NILU7FVTBVDUO72PLGNU3RUSZJOHLYEQDI2QD6RBA, seg_pos={0 0}, upload=561335]: Uploading piece...
Jan 1 08:06:39.267548815 (640h37m39.13863185s) [noise=true , address=145.79.227.44:4213 , piece_id=ENIA5SMSCD7QX6P4ELVIJAFTJIYSZ4RDP7KPKWQ6COKUZF3WXJDQ, seg_pos={0 0}, upload=561335]: Uploading piece...
Jan 1 08:06:39.267565676 (640h37m39.138648721s) [noise=true , address=45.91.116.6:4001 , piece_id=IA43L7PEUXG2F64REP4GZWURKMEJBWZBCYQJXMGDPLZX2FJBYYZQ, seg_pos={0 0}, upload=561335]: Uploading piece...
Jan 1 08:06:39.267812327 (640h37m39.138895372s) [noise=true , address=77.90.31.15:4732 , piece_id=NQ4YN73ZNXTQ3XSTIREDNOFNI4CDX6X4ZLINL2I65BPESTSXNK6A, seg_pos={0 0}, upload=561335]: Uploading piece...
Jan 1 08:06:39.267997900 (640h37m39.139080935s) [noise=true , address=193.149.164.143:4694 , piece_id=JWHH6WDJT2D4SX4Y2DUPJCXVYFHB5ITUANBG63YPDI5KCVFVASDA, seg_pos={0 0}, upload=561335]: Uploading piece...
Jan 1 08:06:39.268112398 (640h37m39.139195443s) [noise=true , address=145.79.227.49:4369 , piece_id=BSVLOVXXRKQROP56TPYDPTV6YRG6WGCTHKER3ZE76VQ3OUQLCG6Q, seg_pos={0 0}, upload=561335]: Uploading piece...
Jan 1 08:06:39.268406639 (640h37m39.139489684s) [noise=true , address=145.79.227.33:4880 , piece_id=WN2T2IFGZPE5ZWVCIU4AVXP4KUIFYWFZ56O5WA3EDVM2UPGDOJ2A, seg_pos={0 0}, upload=561335]: Uploading piece...
Jan 1 08:06:40.656816578 (640h37m40.527899623s) [upload=561335]: Next returned nil segment
Jan 1 08:06:40.656833560 (640h37m40.527916595s) [upload=561335]: Waiting for error group managing segments...
Jan 1 08:06:41.014218117 (640h37m40.885301162s) [noise=true , address=64.49.13.170:4718 , piece_id=MFZMTGAUBC65VN6KKLZDUCXI7RF6QZWGOCOHWJMEBITCXIL6Q5QQ, seg_pos={0 0}, upload=561335]: Done uploading piece. err: <nil>
Jan 1 08:06:41.157461380 (640h37m41.028544425s) [noise=true , address=145.79.227.13:4812 , piece_id=NQXXT7UJSYSOTQOU2TEX3M3RRQS7CJJN5KTUGEKFEXKSH2VY4Q7A, seg_pos={0 0}, upload=561335]: Done uploading piece. err: <nil>
Jan 1 08:06:41.331693451 (640h37m41.202776486s) [noise=true , address=77.90.31.15:4732 , piece_id=NQ4YN73ZNXTQ3XSTIREDNOFNI4CDX6X4ZLINL2I65BPESTSXNK6A, seg_pos={0 0}, upload=561335]: Done uploading piece. err: <nil>
Jan 1 08:06:41.332724885 (640h37m41.20380793s) [noise=true , address=145.79.227.35:4498 , piece_id=MZ6FBJE3D2ZVNO54464FUQ6AN2RCR36Z7YVPOWNB3X25UJFZ7OGQ, seg_pos={0 0}, upload=561335]: Done uploading piece. err: <nil>
Jan 1 08:06:41.382546381 (640h37m41.253629416s) [noise=true , address=77.90.31.12:4610 , piece_id=KRA6U6RAZLXQNOBW6HZL7QLLQK7GJXADHOC3QQH55QIE5T2SI7KA, seg_pos={0 0}, upload=561335]: Done uploading piece. err: <nil>
Jan 1 08:06:41.388091100 (640h37m41.259174135s) [noise=true , address=64.49.13.155:4914 , piece_id=R253V6KYITKWIYUNDNVS3EGVNIR5GKDBYAX67TWLFY67JZPJNKHQ, seg_pos={0 0}, upload=561335]: Done uploading piece. err: <nil>
Jan 1 08:06:41.563625079 (640h37m41.434708114s) [noise=true , address=45.91.116.9:4266 , piece_id=3RUSCCKAC36NILU7FVTBVDUO72PLGNU3RUSZJOHLYEQDI2QD6RBA, seg_pos={0 0}, upload=561335]: Done uploading piece. err: <nil>
Jan 1 08:06:41.911410376 (640h37m41.782493411s) [noise=true , address=193.149.164.143:4694 , piece_id=JWHH6WDJT2D4SX4Y2DUPJCXVYFHB5ITUANBG63YPDI5KCVFVASDA, seg_pos={0 0}, upload=561335]: Done uploading piece. err: <nil>
Jan 1 08:06:42.051037685 (640h37m41.92212073s) [noise=true , address=145.79.227.29:4973 , piece_id=SOSOH3ZHT4OFVIWXS27EPVJFJCI64VTD44HPNISPYKSBBD5ALNAA, seg_pos={0 0}, upload=561335]: Done uploading piece. err: <nil>
Jan 1 08:06:42.084648975 (640h37m41.95573201s) [noise=true , address=145.79.227.44:4213 , piece_id=ENIA5SMSCD7QX6P4ELVIJAFTJIYSZ4RDP7KPKWQ6COKUZF3WXJDQ, seg_pos={0 0}, upload=561335]: Done uploading piece. err: <nil>
Jan 1 08:06:42.096110790 (640h37m41.967193825s) [noise=true , address=145.79.227.49:4369 , piece_id=BSVLOVXXRKQROP56TPYDPTV6YRG6WGCTHKER3ZE76VQ3OUQLCG6Q, seg_pos={0 0}, upload=561335]: Done uploading piece. err: <nil>
Jan 1 08:06:42.104023627 (640h37m41.975106672s) [noise=true , address=145.79.227.30:4776 , piece_id=BLBTAEJLWTL4O2FCB27WERV4FN666F6XT6HXQOD563TYUCVJOLOQ, seg_pos={0 0}, upload=561335]: Done uploading piece. err: <nil>
Jan 1 08:06:42.109017707 (640h37m41.980100762s) [noise=true , address=145.79.227.40:4067 , piece_id=RNGN65VKKQZ3A7OPI2VKA5EBSYCKXMHVQGO6YJXG2YRX5EQEYCQA, seg_pos={0 0}, upload=561335]: Done uploading piece. err: <nil>
Jan 1 08:06:42.150694945 (640h37m42.02177798s) [noise=true , address=145.79.227.43:4182 , piece_id=GXY3PEKWQSHZ2WJHJEF54XU75BD2MHL5XYYRXCTVOKD3BFOFY33A, seg_pos={0 0}, upload=561335]: Done uploading piece. err: <nil>
Jan 1 08:06:42.150713521 (640h37m42.021796556s) [seg_pos={0 0}, upload=561335]: Segment reached optimal threshold of 14 pieces.
Jan 1 08:06:42.150857585 (640h37m42.021940631s) [noise=true , address=64.49.13.169:4716 , piece_id=YZGERUD3ZPPHAGU6MHCIDBSXELDFPFNEQ4A2CKBCZD2TTF7B5O6Q, seg_pos={0 0}, upload=561335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:47710->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:47710->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47710->64.49.13.169:4716: use of closed network connection
Jan 1 08:06:42.150862715 (640h37m42.02194575s) [noise=true , address=64.49.13.169:4716 , piece_id=YZGERUD3ZPPHAGU6MHCIDBSXELDFPFNEQ4A2CKBCZD2TTF7B5O6Q, seg_pos={0 0}, upload=561335]: Long tail cancellation due to optimal threshold
Jan 1 08:06:42.150906138 (640h37m42.021989173s) [noise=true , address=145.79.227.33:4880 , piece_id=WN2T2IFGZPE5ZWVCIU4AVXP4KUIFYWFZ56O5WA3EDVM2UPGDOJ2A, seg_pos={0 0}, upload=561335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sFbwzYfjUGQqrKJhgwDg5dvormM1Xu4yc97awR5kgZUAsKiEn): protocol: expected piece hash; context canceled; EOF
Jan 1 08:06:42.150915375 (640h37m42.02199841s) [noise=true , address=145.79.227.33:4880 , piece_id=WN2T2IFGZPE5ZWVCIU4AVXP4KUIFYWFZ56O5WA3EDVM2UPGDOJ2A, seg_pos={0 0}, upload=561335]: Long tail cancellation due to optimal threshold
Jan 1 08:06:42.151035294 (640h37m42.022118329s) [noise=true , address=45.91.116.6:4001 , piece_id=IA43L7PEUXG2F64REP4GZWURKMEJBWZBCYQJXMGDPLZX2FJBYYZQ, seg_pos={0 0}, upload=561335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQJmv9n2yWXZtCCaABrxkB3NN6vDwLdiBkRibDN3t6YNmcmGP): protocol: expected piece hash; context canceled; EOF
Jan 1 08:06:42.151046475 (640h37m42.02212951s) [noise=true , address=45.91.116.6:4001 , piece_id=IA43L7PEUXG2F64REP4GZWURKMEJBWZBCYQJXMGDPLZX2FJBYYZQ, seg_pos={0 0}, upload=561335]: Long tail cancellation due to optimal threshold
Jan 1 08:06:42.151055602 (640h37m42.022138637s) [noise=true , address=77.90.31.13:4654 , piece_id=JC34MCI3U7Y7Y5JDRYHRU6ACPYQNNJKMDIK3FME5VWMUAUBRFVDA, seg_pos={0 0}, upload=561335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8): protocol: write tcp 188.166.225.170:50358->77.90.31.13:4654: use of closed network connection; write tcp 188.166.225.170:50358->77.90.31.13:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50358->77.90.31.13:4654: use of closed network connection
Jan 1 08:06:42.151061083 (640h37m42.022144118s) [noise=true , address=77.90.31.13:4654 , piece_id=JC34MCI3U7Y7Y5JDRYHRU6ACPYQNNJKMDIK3FME5VWMUAUBRFVDA, seg_pos={0 0}, upload=561335]: Long tail cancellation due to optimal threshold
Jan 1 08:06:42.151075680 (640h37m42.022158716s) [seg_pos={0 0}, upload=561335]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 1 08:06:42.151139893 (640h37m42.022222928s) [upload=561335]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 1 08:06:42.151144381 (640h37m42.022227416s) [upload=561335]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 1 08:06:42.440313737 (640h37m42.311396782s) [upload=561335]: Done uploading segments...
Jan 1 08:06:42.440334847 (640h37m42.311417882s) [upload=561335]: Upload finished. err: <nil>
Duration: 33.156247975s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [.XX..X...X.XX.X...X.........X.X........XX.X...X..X....XXX...] duration:
5.962623268s trace:
https://jaeger.dev.storj.io/trace/20c951f57c07dfa9
FAIL (2026-01-01 08:07:58 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: B3VBW3KAAHNOFEAUALRSFT4NUYDDR335IEZKDYHNHQRAPT4INEXQ) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: 37BBHOW73QYQJDFGK36WPP733ZXTHUXKDSHJGT2JLXXAQEWCALPA) context deadline exceeded
======LOGS======
Jan 1 08:05:33.167157584 (640h36m33.038240619s) [upload=561319]: Starting upload
Jan 1 08:05:33.167395737 (640h36m33.038478782s) [upload=561319]: Done starting upload
Jan 1 08:05:33.167588645 (640h36m33.03867169s) [upload=561319]: Uploading segments...
Jan 1 08:05:33.167794657 (640h36m33.038877693s) [upload=561319]: Got next segment. Inline: false
Jan 1 08:05:33.167935155 (640h36m33.0390182s) [upload=561319]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 1 08:05:33.167959080 (640h36m33.039042115s) [upload=561319]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 1 08:05:33.425006192 (640h36m33.296089237s) [seg_pos={0 0}, upload=561319]: Begin upload segment...
Jan 1 08:05:33.425075253 (640h36m33.296158298s) [seg_pos={0 0}, upload=561319]: Done begin upload segment.
Jan 1 08:05:33.425092566 (640h36m33.296175601s) [noise=true , address=145.79.227.3:4150 , piece_id=RKEABVHWH3CUB7EJG4U3TA6RPQ6M6VRGJQNSME2BPE2FX433CRSQ, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.425116662 (640h36m33.296199707s) [noise=true , address=145.79.227.26:4653 , piece_id=FPTDZAOGL56VZOYAVT2RF5RJSRDLAWEQ5WDBERCLPCDIAGPWJSXA, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.425560297 (640h36m33.296643332s) [noise=true , address=45.91.116.8:4048 , piece_id=TKRTWZRZ5GTOPZBSNYKPODKEGZC3HG5HMRCSCXC56XY6UPR7ICSA, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.425818839 (640h36m33.296901884s) [noise=true , address=145.79.227.15:4873 , piece_id=DA6OMDRTENINDEKH7U7IA2FQ5P25OX7C2U6P5XC2VISHTIPQAWVA, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.426163245 (640h36m33.29724629s) [noise=true , address=145.79.227.41:4085 , piece_id=SBK75752ZTAHALZHLMUSFY6LQ5CFX7M6XE7USIFWQQCWDGKUCO7Q, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.426498062 (640h36m33.297581097s) [noise=true , address=77.90.31.13:4629 , piece_id=B3VBW3KAAHNOFEAUALRSFT4NUYDDR335IEZKDYHNHQRAPT4INEXQ, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.426856224 (640h36m33.29793927s) [noise=true , address=77.90.31.19:4842 , piece_id=L2AOC5E63OF4HGKR7CKCXSPU2OKNBICYNN2MDD4LRNYGNLKVSN4Q, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.427460885 (640h36m33.298543931s) [noise=true , address=145.79.227.9:4367 , piece_id=VF572O4NUNL6V3EVSHMY2FMFTYGFO66BG766NGX34PUE5D6IT5ZQ, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.427862600 (640h36m33.298945645s) [noise=true , address=77.90.31.12:4589 , piece_id=37BBHOW73QYQJDFGK36WPP733ZXTHUXKDSHJGT2JLXXAQEWCALPA, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.428023056 (640h36m33.299106091s) [noise=true , address=45.91.116.7:4009 , piece_id=GJGADK44KRYQRG773AQSBF67BCBBBECTSSHORUYICXQR6SD5E3KQ, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.428268934 (640h36m33.299351979s) [noise=true , address=45.91.116.4:4940 , piece_id=ETQF4GV3O5RCGFINVNORRXBISEKHLSGTKNQLC4O2AHZZKCYOKTHQ, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.428431534 (640h36m33.299514579s) [noise=true , address=145.79.227.4:4187 , piece_id=MONCIHND5Y7OMAPIERB7WX7TVX7NQX6T4OQF54LUPTIF37HHAJOQ, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.428636264 (640h36m33.299719309s) [noise=true , address=45.91.116.6:4975 , piece_id=LR4C5PTPP6B3KUIOEOOGQYNQZOVFUGA4ISRZSXKX6ZAU2PAFQJAQ, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.428918822 (640h36m33.300001867s) [noise=true , address=45.91.116.10:4084 , piece_id=XNOGQNIKCD7PNGHBBKM7CTKXMJ2XKAF3GQ75JUEENK6F6FCS2TOQ, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.429007680 (640h36m33.300090725s) [noise=true , address=77.90.31.16:4735 , piece_id=T2XYKIP6TWORE2OOHCBNMBFS3RGDOCVENFOPFHIMRMGN34IZ7GWA, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.429386582 (640h36m33.300469617s) [noise=true , address=77.90.31.11:4553 , piece_id=M7V5WCD4FYNJLQIQCZHB4F3LGTPNIQDWVM7XH5X7YKL6RZHVWCCA, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:35.136600720 (640h36m35.007683765s) [upload=561319]: Next returned nil segment
Jan 1 08:05:35.136614055 (640h36m35.00769709s) [upload=561319]: Waiting for error group managing segments...
Jan 1 08:05:35.418160256 (640h36m35.289243301s) [noise=true , address=145.79.227.26:4653 , piece_id=FPTDZAOGL56VZOYAVT2RF5RJSRDLAWEQ5WDBERCLPCDIAGPWJSXA, seg_pos={0 0}, upload=561319]: Done uploading piece. err: <nil>
Jan 1 08:05:35.434506661 (640h36m35.305589696s) [noise=true , address=145.79.227.9:4367 , piece_id=VF572O4NUNL6V3EVSHMY2FMFTYGFO66BG766NGX34PUE5D6IT5ZQ, seg_pos={0 0}, upload=561319]: Done uploading piece. err: <nil>
Jan 1 08:05:35.445590456 (640h36m35.316673501s) [noise=true , address=145.79.227.15:4873 , piece_id=DA6OMDRTENINDEKH7U7IA2FQ5P25OX7C2U6P5XC2VISHTIPQAWVA, seg_pos={0 0}, upload=561319]: Done uploading piece. err: <nil>
Jan 1 08:05:35.456856528 (640h36m35.327939563s) [noise=true , address=145.79.227.4:4187 , piece_id=MONCIHND5Y7OMAPIERB7WX7TVX7NQX6T4OQF54LUPTIF37HHAJOQ, seg_pos={0 0}, upload=561319]: Done uploading piece. err: <nil>
Jan 1 08:05:35.501865411 (640h36m35.372948446s) [noise=true , address=145.79.227.41:4085 , piece_id=SBK75752ZTAHALZHLMUSFY6LQ5CFX7M6XE7USIFWQQCWDGKUCO7Q, seg_pos={0 0}, upload=561319]: Done uploading piece. err: <nil>
Jan 1 08:05:35.507303396 (640h36m35.378386431s) [noise=true , address=145.79.227.3:4150 , piece_id=RKEABVHWH3CUB7EJG4U3TA6RPQ6M6VRGJQNSME2BPE2FX433CRSQ, seg_pos={0 0}, upload=561319]: Done uploading piece. err: <nil>
Jan 1 08:05:35.602364924 (640h36m35.473447959s) [noise=true , address=77.90.31.19:4842 , piece_id=L2AOC5E63OF4HGKR7CKCXSPU2OKNBICYNN2MDD4LRNYGNLKVSN4Q, seg_pos={0 0}, upload=561319]: Done uploading piece. err: <nil>
Jan 1 08:05:35.609000360 (640h36m35.480083405s) [noise=true , address=77.90.31.13:4629 , piece_id=B3VBW3KAAHNOFEAUALRSFT4NUYDDR335IEZKDYHNHQRAPT4INEXQ, seg_pos={0 0}, upload=561319]: Done uploading piece. err: <nil>
Jan 1 08:05:35.709299663 (640h36m35.580382698s) [noise=true , address=77.90.31.16:4735 , piece_id=T2XYKIP6TWORE2OOHCBNMBFS3RGDOCVENFOPFHIMRMGN34IZ7GWA, seg_pos={0 0}, upload=561319]: Done uploading piece. err: <nil>
Jan 1 08:05:35.711602518 (640h36m35.582685563s) [noise=true , address=77.90.31.12:4589 , piece_id=37BBHOW73QYQJDFGK36WPP733ZXTHUXKDSHJGT2JLXXAQEWCALPA, seg_pos={0 0}, upload=561319]: Done uploading piece. err: <nil>
Jan 1 08:05:35.720153349 (640h36m35.591236384s) [noise=true , address=45.91.116.6:4975 , piece_id=LR4C5PTPP6B3KUIOEOOGQYNQZOVFUGA4ISRZSXKX6ZAU2PAFQJAQ, seg_pos={0 0}, upload=561319]: Done uploading piece. err: <nil>
Jan 1 08:05:35.735457540 (640h36m35.606540576s) [noise=true , address=77.90.31.11:4553 , piece_id=M7V5WCD4FYNJLQIQCZHB4F3LGTPNIQDWVM7XH5X7YKL6RZHVWCCA, seg_pos={0 0}, upload=561319]: Done uploading piece. err: <nil>
Jan 1 08:05:35.850795468 (640h36m35.721878504s) [noise=true , address=45.91.116.4:4940 , piece_id=ETQF4GV3O5RCGFINVNORRXBISEKHLSGTKNQLC4O2AHZZKCYOKTHQ, seg_pos={0 0}, upload=561319]: Done uploading piece. err: <nil>
Jan 1 08:05:35.850814644 (640h36m35.72189769s) [seg_pos={0 0}, upload=561319]: Segment reached optimal threshold of 13 pieces.
Jan 1 08:05:35.850994106 (640h36m35.722077151s) [noise=true , address=45.91.116.10:4084 , piece_id=XNOGQNIKCD7PNGHBBKM7CTKXMJ2XKAF3GQ75JUEENK6F6FCS2TOQ, seg_pos={0 0}, upload=561319]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF
Jan 1 08:05:35.851000668 (640h36m35.722083713s) [noise=true , address=45.91.116.10:4084 , piece_id=XNOGQNIKCD7PNGHBBKM7CTKXMJ2XKAF3GQ75JUEENK6F6FCS2TOQ, seg_pos={0 0}, upload=561319]: Long tail cancellation due to optimal threshold
Jan 1 08:05:35.851128392 (640h36m35.722211427s) [noise=true , address=45.91.116.7:4009 , piece_id=GJGADK44KRYQRG773AQSBF67BCBBBECTSSHORUYICXQR6SD5E3KQ, seg_pos={0 0}, upload=561319]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Jan 1 08:05:35.851133912 (640h36m35.722216957s) [noise=true , address=45.91.116.7:4009 , piece_id=GJGADK44KRYQRG773AQSBF67BCBBBECTSSHORUYICXQR6SD5E3KQ, seg_pos={0 0}, upload=561319]: Long tail cancellation due to optimal threshold
Jan 1 08:05:35.851382736 (640h36m35.722465781s) [noise=true , address=45.91.116.8:4048 , piece_id=TKRTWZRZ5GTOPZBSNYKPODKEGZC3HG5HMRCSCXC56XY6UPR7ICSA, seg_pos={0 0}, upload=561319]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:36492->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:36492->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36492->45.91.116.8:4048: use of closed network connection
Jan 1 08:05:35.851402664 (640h36m35.722485709s) [noise=true , address=45.91.116.8:4048 , piece_id=TKRTWZRZ5GTOPZBSNYKPODKEGZC3HG5HMRCSCXC56XY6UPR7ICSA, seg_pos={0 0}, upload=561319]: Long tail cancellation due to optimal threshold
Jan 1 08:05:35.851460284 (640h36m35.722543329s) [seg_pos={0 0}, upload=561319]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 1 08:05:35.851566416 (640h36m35.722649471s) [upload=561319]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 1 08:05:35.851573239 (640h36m35.722656274s) [upload=561319]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 1 08:05:36.144226899 (640h36m36.015309944s) [upload=561319]: Done uploading segments...
Jan 1 08:05:36.144246647 (640h36m36.015329692s) [upload=561319]: Upload finished. err: <nil>
Duration: 5.962651221s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................X.......................] duration:
12.323748883s trace:
https://jaeger.dev.storj.io/trace/44e9d74e3b154e8e
FAIL (2026-01-01 08:07:47 UTC)
orbiter warning: context deadline exceeded
Duration: 12.323772337s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
7 out of 60 is failed (threshold=3) [...........X.....X..XX.........................XX..........X] duration:
1m0.004157836s trace:
https://jaeger.dev.storj.io/trace/e95ca73698c6b71
FAIL (2026-01-01 08:07:38 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
.227.43:4182 , piece_id=GXY3PEKWQSHZ2WJHJEF54XU75BD2MHL5XYYRXCTVOKD3BFOFY33A, seg_pos={0 0}, upload=561335]: Uploading piece...
Jan 1 08:06:39.265949369 (640h37m39.137032404s) [noise=true , address=64.49.13.170:4718 , piece_id=MFZMTGAUBC65VN6KKLZDUCXI7RF6QZWGOCOHWJMEBITCXIL6Q5QQ, seg_pos={0 0}, upload=561335]: Uploading piece...
Jan 1 08:06:39.266002000 (640h37m39.137085045s) [noise=true , address=77.90.31.13:4654 , piece_id=JC34MCI3U7Y7Y5JDRYHRU6ACPYQNNJKMDIK3FME5VWMUAUBRFVDA, seg_pos={0 0}, upload=561335]: Uploading piece...
Jan 1 08:06:39.266075890 (640h37m39.137158925s) [noise=true , address=145.79.227.35:4498 , piece_id=MZ6FBJE3D2ZVNO54464FUQ6AN2RCR36Z7YVPOWNB3X25UJFZ7OGQ, seg_pos={0 0}, upload=561335]: Uploading piece...
Jan 1 08:06:39.266104886 (640h37m39.137187931s) [noise=true , address=145.79.227.40:4067 , piece_id=RNGN65VKKQZ3A7OPI2VKA5EBSYCKXMHVQGO6YJXG2YRX5EQEYCQA, seg_pos={0 0}, upload=561335]: Uploading piece...
Jan 1 08:06:39.266729564 (640h37m39.137812599s) [noise=true , address=77.90.31.12:4610 , piece_id=KRA6U6RAZLXQNOBW6HZL7QLLQK7GJXADHOC3QQH55QIE5T2SI7KA, seg_pos={0 0}, upload=561335]: Uploading piece...
Jan 1 08:06:39.266817723 (640h37m39.137900758s) [noise=true , address=145.79.227.30:4776 , piece_id=BLBTAEJLWTL4O2FCB27WERV4FN666F6XT6HXQOD563TYUCVJOLOQ, seg_pos={0 0}, upload=561335]: Uploading piece...
Jan 1 08:06:39.267202215 (640h37m39.13828526s) [noise=true , address=145.79.227.29:4973 , piece_id=SOSOH3ZHT4OFVIWXS27EPVJFJCI64VTD44HPNISPYKSBBD5ALNAA, seg_pos={0 0}, upload=561335]: Uploading piece...
Jan 1 08:06:39.267270585 (640h37m39.13835363s) [noise=true , address=45.91.116.9:4266 , piece_id=3RUSCCKAC36NILU7FVTBVDUO72PLGNU3RUSZJOHLYEQDI2QD6RBA, seg_pos={0 0}, upload=561335]: Uploading piece...
Jan 1 08:06:39.267548815 (640h37m39.13863185s) [noise=true , address=145.79.227.44:4213 , piece_id=ENIA5SMSCD7QX6P4ELVIJAFTJIYSZ4RDP7KPKWQ6COKUZF3WXJDQ, seg_pos={0 0}, upload=561335]: Uploading piece...
Jan 1 08:06:39.267565676 (640h37m39.138648721s) [noise=true , address=45.91.116.6:4001 , piece_id=IA43L7PEUXG2F64REP4GZWURKMEJBWZBCYQJXMGDPLZX2FJBYYZQ, seg_pos={0 0}, upload=561335]: Uploading piece...
Jan 1 08:06:39.267812327 (640h37m39.138895372s) [noise=true , address=77.90.31.15:4732 , piece_id=NQ4YN73ZNXTQ3XSTIREDNOFNI4CDX6X4ZLINL2I65BPESTSXNK6A, seg_pos={0 0}, upload=561335]: Uploading piece...
Jan 1 08:06:39.267997900 (640h37m39.139080935s) [noise=true , address=193.149.164.143:4694 , piece_id=JWHH6WDJT2D4SX4Y2DUPJCXVYFHB5ITUANBG63YPDI5KCVFVASDA, seg_pos={0 0}, upload=561335]: Uploading piece...
Jan 1 08:06:39.268112398 (640h37m39.139195443s) [noise=true , address=145.79.227.49:4369 , piece_id=BSVLOVXXRKQROP56TPYDPTV6YRG6WGCTHKER3ZE76VQ3OUQLCG6Q, seg_pos={0 0}, upload=561335]: Uploading piece...
Jan 1 08:06:39.268406639 (640h37m39.139489684s) [noise=true , address=145.79.227.33:4880 , piece_id=WN2T2IFGZPE5ZWVCIU4AVXP4KUIFYWFZ56O5WA3EDVM2UPGDOJ2A, seg_pos={0 0}, upload=561335]: Uploading piece...
Jan 1 08:06:40.656816578 (640h37m40.527899623s) [upload=561335]: Next returned nil segment
Jan 1 08:06:40.656833560 (640h37m40.527916595s) [upload=561335]: Waiting for error group managing segments...
Jan 1 08:06:41.014218117 (640h37m40.885301162s) [noise=true , address=64.49.13.170:4718 , piece_id=MFZMTGAUBC65VN6KKLZDUCXI7RF6QZWGOCOHWJMEBITCXIL6Q5QQ, seg_pos={0 0}, upload=561335]: Done uploading piece. err: <nil>
Jan 1 08:06:41.157461380 (640h37m41.028544425s) [noise=true , address=145.79.227.13:4812 , piece_id=NQXXT7UJSYSOTQOU2TEX3M3RRQS7CJJN5KTUGEKFEXKSH2VY4Q7A, seg_pos={0 0}, upload=561335]: Done uploading piece. err: <nil>
Jan 1 08:06:41.331693451 (640h37m41.202776486s) [noise=true , address=77.90.31.15:4732 , piece_id=NQ4YN73ZNXTQ3XSTIREDNOFNI4CDX6X4ZLINL2I65BPESTSXNK6A, seg_pos={0 0}, upload=561335]: Done uploading piece. err: <nil>
Jan 1 08:06:41.332724885 (640h37m41.20380793s) [noise=true , address=145.79.227.35:4498 , piece_id=MZ6FBJE3D2ZVNO54464FUQ6AN2RCR36Z7YVPOWNB3X25UJFZ7OGQ, seg_pos={0 0}, upload=561335]: Done uploading piece. err: <nil>
Jan 1 08:06:41.382546381 (640h37m41.253629416s) [noise=true , address=77.90.31.12:4610 , piece_id=KRA6U6RAZLXQNOBW6HZL7QLLQK7GJXADHOC3QQH55QIE5T2SI7KA, seg_pos={0 0}, upload=561335]: Done uploading piece. err: <nil>
Jan 1 08:06:41.388091100 (640h37m41.259174135s) [noise=true , address=64.49.13.155:4914 , piece_id=R253V6KYITKWIYUNDNVS3EGVNIR5GKDBYAX67TWLFY67JZPJNKHQ, seg_pos={0 0}, upload=561335]: Done uploading piece. err: <nil>
Jan 1 08:06:41.563625079 (640h37m41.434708114s) [noise=true , address=45.91.116.9:4266 , piece_id=3RUSCCKAC36NILU7FVTBVDUO72PLGNU3RUSZJOHLYEQDI2QD6RBA, seg_pos={0 0}, upload=561335]: Done uploading piece. err: <nil>
Jan 1 08:06:41.911410376 (640h37m41.782493411s) [noise=true , address=193.149.164.143:4694 , piece_id=JWHH6WDJT2D4SX4Y2DUPJCXVYFHB5ITUANBG63YPDI5KCVFVASDA, seg_pos={0 0}, upload=561335]: Done uploading piece. err: <nil>
Jan 1 08:06:42.051037685 (640h37m41.92212073s) [noise=true , address=145.79.227.29:4973 , piece_id=SOSOH3ZHT4OFVIWXS27EPVJFJCI64VTD44HPNISPYKSBBD5ALNAA, seg_pos={0 0}, upload=561335]: Done uploading piece. err: <nil>
Jan 1 08:06:42.084648975 (640h37m41.95573201s) [noise=true , address=145.79.227.44:4213 , piece_id=ENIA5SMSCD7QX6P4ELVIJAFTJIYSZ4RDP7KPKWQ6COKUZF3WXJDQ, seg_pos={0 0}, upload=561335]: Done uploading piece. err: <nil>
Jan 1 08:06:42.096110790 (640h37m41.967193825s) [noise=true , address=145.79.227.49:4369 , piece_id=BSVLOVXXRKQROP56TPYDPTV6YRG6WGCTHKER3ZE76VQ3OUQLCG6Q, seg_pos={0 0}, upload=561335]: Done uploading piece. err: <nil>
Jan 1 08:06:42.104023627 (640h37m41.975106672s) [noise=true , address=145.79.227.30:4776 , piece_id=BLBTAEJLWTL4O2FCB27WERV4FN666F6XT6HXQOD563TYUCVJOLOQ, seg_pos={0 0}, upload=561335]: Done uploading piece. err: <nil>
Jan 1 08:06:42.109017707 (640h37m41.980100762s) [noise=true , address=145.79.227.40:4067 , piece_id=RNGN65VKKQZ3A7OPI2VKA5EBSYCKXMHVQGO6YJXG2YRX5EQEYCQA, seg_pos={0 0}, upload=561335]: Done uploading piece. err: <nil>
Jan 1 08:06:42.150694945 (640h37m42.02177798s) [noise=true , address=145.79.227.43:4182 , piece_id=GXY3PEKWQSHZ2WJHJEF54XU75BD2MHL5XYYRXCTVOKD3BFOFY33A, seg_pos={0 0}, upload=561335]: Done uploading piece. err: <nil>
Jan 1 08:06:42.150713521 (640h37m42.021796556s) [seg_pos={0 0}, upload=561335]: Segment reached optimal threshold of 14 pieces.
Jan 1 08:06:42.150857585 (640h37m42.021940631s) [noise=true , address=64.49.13.169:4716 , piece_id=YZGERUD3ZPPHAGU6MHCIDBSXELDFPFNEQ4A2CKBCZD2TTF7B5O6Q, seg_pos={0 0}, upload=561335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:47710->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:47710->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47710->64.49.13.169:4716: use of closed network connection
Jan 1 08:06:42.150862715 (640h37m42.02194575s) [noise=true , address=64.49.13.169:4716 , piece_id=YZGERUD3ZPPHAGU6MHCIDBSXELDFPFNEQ4A2CKBCZD2TTF7B5O6Q, seg_pos={0 0}, upload=561335]: Long tail cancellation due to optimal threshold
Jan 1 08:06:42.150906138 (640h37m42.021989173s) [noise=true , address=145.79.227.33:4880 , piece_id=WN2T2IFGZPE5ZWVCIU4AVXP4KUIFYWFZ56O5WA3EDVM2UPGDOJ2A, seg_pos={0 0}, upload=561335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1sFbwzYfjUGQqrKJhgwDg5dvormM1Xu4yc97awR5kgZUAsKiEn): protocol: expected piece hash; context canceled; EOF
Jan 1 08:06:42.150915375 (640h37m42.02199841s) [noise=true , address=145.79.227.33:4880 , piece_id=WN2T2IFGZPE5ZWVCIU4AVXP4KUIFYWFZ56O5WA3EDVM2UPGDOJ2A, seg_pos={0 0}, upload=561335]: Long tail cancellation due to optimal threshold
Jan 1 08:06:42.151035294 (640h37m42.022118329s) [noise=true , address=45.91.116.6:4001 , piece_id=IA43L7PEUXG2F64REP4GZWURKMEJBWZBCYQJXMGDPLZX2FJBYYZQ, seg_pos={0 0}, upload=561335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQJmv9n2yWXZtCCaABrxkB3NN6vDwLdiBkRibDN3t6YNmcmGP): protocol: expected piece hash; context canceled; EOF
Jan 1 08:06:42.151046475 (640h37m42.02212951s) [noise=true , address=45.91.116.6:4001 , piece_id=IA43L7PEUXG2F64REP4GZWURKMEJBWZBCYQJXMGDPLZX2FJBYYZQ, seg_pos={0 0}, upload=561335]: Long tail cancellation due to optimal threshold
Jan 1 08:06:42.151055602 (640h37m42.022138637s) [noise=true , address=77.90.31.13:4654 , piece_id=JC34MCI3U7Y7Y5JDRYHRU6ACPYQNNJKMDIK3FME5VWMUAUBRFVDA, seg_pos={0 0}, upload=561335]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1227esgYd6VByyMSV6ZBcvYqx6PahMt7SUgvYzYmiWJavNXGza8): protocol: write tcp 188.166.225.170:50358->77.90.31.13:4654: use of closed network connection; write tcp 188.166.225.170:50358->77.90.31.13:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:50358->77.90.31.13:4654: use of closed network connection
Jan 1 08:06:42.151061083 (640h37m42.022144118s) [noise=true , address=77.90.31.13:4654 , piece_id=JC34MCI3U7Y7Y5JDRYHRU6ACPYQNNJKMDIK3FME5VWMUAUBRFVDA, seg_pos={0 0}, upload=561335]: Long tail cancellation due to optimal threshold
Jan 1 08:06:42.151075680 (640h37m42.022158716s) [seg_pos={0 0}, upload=561335]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 1 08:06:42.151139893 (640h37m42.022222928s) [upload=561335]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 1 08:06:42.151144381 (640h37m42.022227416s) [upload=561335]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 1 08:06:42.440313737 (640h37m42.311396782s) [upload=561335]: Done uploading segments...
Jan 1 08:06:42.440334847 (640h37m42.311417882s) [upload=561335]: Upload finished. err: <nil>
Duration: 1m0.004183987s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
40 out of 60 is failed (threshold=6) [X.XXXXX.XX..XXXXX.XXXX..XX.XXX.XX...X.X..X...XXXXXX..XXXXXXX] duration:
5.001574136s trace:
https://jaeger.dev.storj.io/trace/18db0c1415df8ad9
FAIL (2026-01-01 08:07:23 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001601518s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
39 out of 60 is failed (threshold=6) [.X.XXXXX.XX..XXXXX.XXXX..XX.XXX.XX...X.X..X...XXXXXX..XXXXXX] duration:
5.002092579s trace:
https://jaeger.dev.storj.io/trace/be668214a1b622
FAIL (2026-01-01 08:07:03 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.002117196s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [X.XX..X...X.XX.X...X.........X.X........XX.X...X..X....XXX..] duration:
6.161628735s trace:
https://jaeger.dev.storj.io/trace/20c951f57c07dfa9
FAIL (2026-01-01 08:06:59 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: B3VBW3KAAHNOFEAUALRSFT4NUYDDR335IEZKDYHNHQRAPT4INEXQ) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: 37BBHOW73QYQJDFGK36WPP733ZXTHUXKDSHJGT2JLXXAQEWCALPA) context deadline exceeded
======LOGS======
Jan 1 08:05:33.167157584 (640h36m33.038240619s) [upload=561319]: Starting upload
Jan 1 08:05:33.167395737 (640h36m33.038478782s) [upload=561319]: Done starting upload
Jan 1 08:05:33.167588645 (640h36m33.03867169s) [upload=561319]: Uploading segments...
Jan 1 08:05:33.167794657 (640h36m33.038877693s) [upload=561319]: Got next segment. Inline: false
Jan 1 08:05:33.167935155 (640h36m33.0390182s) [upload=561319]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 1 08:05:33.167959080 (640h36m33.039042115s) [upload=561319]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 1 08:05:33.425006192 (640h36m33.296089237s) [seg_pos={0 0}, upload=561319]: Begin upload segment...
Jan 1 08:05:33.425075253 (640h36m33.296158298s) [seg_pos={0 0}, upload=561319]: Done begin upload segment.
Jan 1 08:05:33.425092566 (640h36m33.296175601s) [noise=true , address=145.79.227.3:4150 , piece_id=RKEABVHWH3CUB7EJG4U3TA6RPQ6M6VRGJQNSME2BPE2FX433CRSQ, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.425116662 (640h36m33.296199707s) [noise=true , address=145.79.227.26:4653 , piece_id=FPTDZAOGL56VZOYAVT2RF5RJSRDLAWEQ5WDBERCLPCDIAGPWJSXA, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.425560297 (640h36m33.296643332s) [noise=true , address=45.91.116.8:4048 , piece_id=TKRTWZRZ5GTOPZBSNYKPODKEGZC3HG5HMRCSCXC56XY6UPR7ICSA, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.425818839 (640h36m33.296901884s) [noise=true , address=145.79.227.15:4873 , piece_id=DA6OMDRTENINDEKH7U7IA2FQ5P25OX7C2U6P5XC2VISHTIPQAWVA, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.426163245 (640h36m33.29724629s) [noise=true , address=145.79.227.41:4085 , piece_id=SBK75752ZTAHALZHLMUSFY6LQ5CFX7M6XE7USIFWQQCWDGKUCO7Q, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.426498062 (640h36m33.297581097s) [noise=true , address=77.90.31.13:4629 , piece_id=B3VBW3KAAHNOFEAUALRSFT4NUYDDR335IEZKDYHNHQRAPT4INEXQ, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.426856224 (640h36m33.29793927s) [noise=true , address=77.90.31.19:4842 , piece_id=L2AOC5E63OF4HGKR7CKCXSPU2OKNBICYNN2MDD4LRNYGNLKVSN4Q, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.427460885 (640h36m33.298543931s) [noise=true , address=145.79.227.9:4367 , piece_id=VF572O4NUNL6V3EVSHMY2FMFTYGFO66BG766NGX34PUE5D6IT5ZQ, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.427862600 (640h36m33.298945645s) [noise=true , address=77.90.31.12:4589 , piece_id=37BBHOW73QYQJDFGK36WPP733ZXTHUXKDSHJGT2JLXXAQEWCALPA, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.428023056 (640h36m33.299106091s) [noise=true , address=45.91.116.7:4009 , piece_id=GJGADK44KRYQRG773AQSBF67BCBBBECTSSHORUYICXQR6SD5E3KQ, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.428268934 (640h36m33.299351979s) [noise=true , address=45.91.116.4:4940 , piece_id=ETQF4GV3O5RCGFINVNORRXBISEKHLSGTKNQLC4O2AHZZKCYOKTHQ, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.428431534 (640h36m33.299514579s) [noise=true , address=145.79.227.4:4187 , piece_id=MONCIHND5Y7OMAPIERB7WX7TVX7NQX6T4OQF54LUPTIF37HHAJOQ, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.428636264 (640h36m33.299719309s) [noise=true , address=45.91.116.6:4975 , piece_id=LR4C5PTPP6B3KUIOEOOGQYNQZOVFUGA4ISRZSXKX6ZAU2PAFQJAQ, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.428918822 (640h36m33.300001867s) [noise=true , address=45.91.116.10:4084 , piece_id=XNOGQNIKCD7PNGHBBKM7CTKXMJ2XKAF3GQ75JUEENK6F6FCS2TOQ, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.429007680 (640h36m33.300090725s) [noise=true , address=77.90.31.16:4735 , piece_id=T2XYKIP6TWORE2OOHCBNMBFS3RGDOCVENFOPFHIMRMGN34IZ7GWA, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.429386582 (640h36m33.300469617s) [noise=true , address=77.90.31.11:4553 , piece_id=M7V5WCD4FYNJLQIQCZHB4F3LGTPNIQDWVM7XH5X7YKL6RZHVWCCA, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:35.136600720 (640h36m35.007683765s) [upload=561319]: Next returned nil segment
Jan 1 08:05:35.136614055 (640h36m35.00769709s) [upload=561319]: Waiting for error group managing segments...
Jan 1 08:05:35.418160256 (640h36m35.289243301s) [noise=true , address=145.79.227.26:4653 , piece_id=FPTDZAOGL56VZOYAVT2RF5RJSRDLAWEQ5WDBERCLPCDIAGPWJSXA, seg_pos={0 0}, upload=561319]: Done uploading piece. err: <nil>
Jan 1 08:05:35.434506661 (640h36m35.305589696s) [noise=true , address=145.79.227.9:4367 , piece_id=VF572O4NUNL6V3EVSHMY2FMFTYGFO66BG766NGX34PUE5D6IT5ZQ, seg_pos={0 0}, upload=561319]: Done uploading piece. err: <nil>
Jan 1 08:05:35.445590456 (640h36m35.316673501s) [noise=true , address=145.79.227.15:4873 , piece_id=DA6OMDRTENINDEKH7U7IA2FQ5P25OX7C2U6P5XC2VISHTIPQAWVA, seg_pos={0 0}, upload=561319]: Done uploading piece. err: <nil>
Jan 1 08:05:35.456856528 (640h36m35.327939563s) [noise=true , address=145.79.227.4:4187 , piece_id=MONCIHND5Y7OMAPIERB7WX7TVX7NQX6T4OQF54LUPTIF37HHAJOQ, seg_pos={0 0}, upload=561319]: Done uploading piece. err: <nil>
Jan 1 08:05:35.501865411 (640h36m35.372948446s) [noise=true , address=145.79.227.41:4085 , piece_id=SBK75752ZTAHALZHLMUSFY6LQ5CFX7M6XE7USIFWQQCWDGKUCO7Q, seg_pos={0 0}, upload=561319]: Done uploading piece. err: <nil>
Jan 1 08:05:35.507303396 (640h36m35.378386431s) [noise=true , address=145.79.227.3:4150 , piece_id=RKEABVHWH3CUB7EJG4U3TA6RPQ6M6VRGJQNSME2BPE2FX433CRSQ, seg_pos={0 0}, upload=561319]: Done uploading piece. err: <nil>
Jan 1 08:05:35.602364924 (640h36m35.473447959s) [noise=true , address=77.90.31.19:4842 , piece_id=L2AOC5E63OF4HGKR7CKCXSPU2OKNBICYNN2MDD4LRNYGNLKVSN4Q, seg_pos={0 0}, upload=561319]: Done uploading piece. err: <nil>
Jan 1 08:05:35.609000360 (640h36m35.480083405s) [noise=true , address=77.90.31.13:4629 , piece_id=B3VBW3KAAHNOFEAUALRSFT4NUYDDR335IEZKDYHNHQRAPT4INEXQ, seg_pos={0 0}, upload=561319]: Done uploading piece. err: <nil>
Jan 1 08:05:35.709299663 (640h36m35.580382698s) [noise=true , address=77.90.31.16:4735 , piece_id=T2XYKIP6TWORE2OOHCBNMBFS3RGDOCVENFOPFHIMRMGN34IZ7GWA, seg_pos={0 0}, upload=561319]: Done uploading piece. err: <nil>
Jan 1 08:05:35.711602518 (640h36m35.582685563s) [noise=true , address=77.90.31.12:4589 , piece_id=37BBHOW73QYQJDFGK36WPP733ZXTHUXKDSHJGT2JLXXAQEWCALPA, seg_pos={0 0}, upload=561319]: Done uploading piece. err: <nil>
Jan 1 08:05:35.720153349 (640h36m35.591236384s) [noise=true , address=45.91.116.6:4975 , piece_id=LR4C5PTPP6B3KUIOEOOGQYNQZOVFUGA4ISRZSXKX6ZAU2PAFQJAQ, seg_pos={0 0}, upload=561319]: Done uploading piece. err: <nil>
Jan 1 08:05:35.735457540 (640h36m35.606540576s) [noise=true , address=77.90.31.11:4553 , piece_id=M7V5WCD4FYNJLQIQCZHB4F3LGTPNIQDWVM7XH5X7YKL6RZHVWCCA, seg_pos={0 0}, upload=561319]: Done uploading piece. err: <nil>
Jan 1 08:05:35.850795468 (640h36m35.721878504s) [noise=true , address=45.91.116.4:4940 , piece_id=ETQF4GV3O5RCGFINVNORRXBISEKHLSGTKNQLC4O2AHZZKCYOKTHQ, seg_pos={0 0}, upload=561319]: Done uploading piece. err: <nil>
Jan 1 08:05:35.850814644 (640h36m35.72189769s) [seg_pos={0 0}, upload=561319]: Segment reached optimal threshold of 13 pieces.
Jan 1 08:05:35.850994106 (640h36m35.722077151s) [noise=true , address=45.91.116.10:4084 , piece_id=XNOGQNIKCD7PNGHBBKM7CTKXMJ2XKAF3GQ75JUEENK6F6FCS2TOQ, seg_pos={0 0}, upload=561319]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF
Jan 1 08:05:35.851000668 (640h36m35.722083713s) [noise=true , address=45.91.116.10:4084 , piece_id=XNOGQNIKCD7PNGHBBKM7CTKXMJ2XKAF3GQ75JUEENK6F6FCS2TOQ, seg_pos={0 0}, upload=561319]: Long tail cancellation due to optimal threshold
Jan 1 08:05:35.851128392 (640h36m35.722211427s) [noise=true , address=45.91.116.7:4009 , piece_id=GJGADK44KRYQRG773AQSBF67BCBBBECTSSHORUYICXQR6SD5E3KQ, seg_pos={0 0}, upload=561319]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Jan 1 08:05:35.851133912 (640h36m35.722216957s) [noise=true , address=45.91.116.7:4009 , piece_id=GJGADK44KRYQRG773AQSBF67BCBBBECTSSHORUYICXQR6SD5E3KQ, seg_pos={0 0}, upload=561319]: Long tail cancellation due to optimal threshold
Jan 1 08:05:35.851382736 (640h36m35.722465781s) [noise=true , address=45.91.116.8:4048 , piece_id=TKRTWZRZ5GTOPZBSNYKPODKEGZC3HG5HMRCSCXC56XY6UPR7ICSA, seg_pos={0 0}, upload=561319]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:36492->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:36492->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36492->45.91.116.8:4048: use of closed network connection
Jan 1 08:05:35.851402664 (640h36m35.722485709s) [noise=true , address=45.91.116.8:4048 , piece_id=TKRTWZRZ5GTOPZBSNYKPODKEGZC3HG5HMRCSCXC56XY6UPR7ICSA, seg_pos={0 0}, upload=561319]: Long tail cancellation due to optimal threshold
Jan 1 08:05:35.851460284 (640h36m35.722543329s) [seg_pos={0 0}, upload=561319]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 1 08:05:35.851566416 (640h36m35.722649471s) [upload=561319]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 1 08:05:35.851573239 (640h36m35.722656274s) [upload=561319]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 1 08:05:36.144226899 (640h36m36.015309944s) [upload=561319]: Done uploading segments...
Jan 1 08:05:36.144246647 (640h36m36.015329692s) [upload=561319]: Upload finished. err: <nil>
Duration: 6.1617105s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
39 out of 60 is failed (threshold=6) [X.X.XXXXX.XX..XXXXX.XXXX..XX.XXX.XX...X.X..X...XXXXXX..XXXXX] duration:
5.001224776s trace:
https://jaeger.dev.storj.io/trace/4e9f4c2031686f4
FAIL (2026-01-01 08:06:44 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001247619s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
20 out of 60 is failed (threshold=3) [XX.XX..X...X.XX.X...X.........X.X........XX.X...X..X....XXX.] duration:
5.934177234s trace:
https://jaeger.dev.storj.io/trace/20c951f57c07dfa9
FAIL (2026-01-01 08:06:31 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: B3VBW3KAAHNOFEAUALRSFT4NUYDDR335IEZKDYHNHQRAPT4INEXQ) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: 37BBHOW73QYQJDFGK36WPP733ZXTHUXKDSHJGT2JLXXAQEWCALPA) context deadline exceeded
======LOGS======
Jan 1 08:05:33.167157584 (640h36m33.038240619s) [upload=561319]: Starting upload
Jan 1 08:05:33.167395737 (640h36m33.038478782s) [upload=561319]: Done starting upload
Jan 1 08:05:33.167588645 (640h36m33.03867169s) [upload=561319]: Uploading segments...
Jan 1 08:05:33.167794657 (640h36m33.038877693s) [upload=561319]: Got next segment. Inline: false
Jan 1 08:05:33.167935155 (640h36m33.0390182s) [upload=561319]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 1 08:05:33.167959080 (640h36m33.039042115s) [upload=561319]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 1 08:05:33.425006192 (640h36m33.296089237s) [seg_pos={0 0}, upload=561319]: Begin upload segment...
Jan 1 08:05:33.425075253 (640h36m33.296158298s) [seg_pos={0 0}, upload=561319]: Done begin upload segment.
Jan 1 08:05:33.425092566 (640h36m33.296175601s) [noise=true , address=145.79.227.3:4150 , piece_id=RKEABVHWH3CUB7EJG4U3TA6RPQ6M6VRGJQNSME2BPE2FX433CRSQ, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.425116662 (640h36m33.296199707s) [noise=true , address=145.79.227.26:4653 , piece_id=FPTDZAOGL56VZOYAVT2RF5RJSRDLAWEQ5WDBERCLPCDIAGPWJSXA, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.425560297 (640h36m33.296643332s) [noise=true , address=45.91.116.8:4048 , piece_id=TKRTWZRZ5GTOPZBSNYKPODKEGZC3HG5HMRCSCXC56XY6UPR7ICSA, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.425818839 (640h36m33.296901884s) [noise=true , address=145.79.227.15:4873 , piece_id=DA6OMDRTENINDEKH7U7IA2FQ5P25OX7C2U6P5XC2VISHTIPQAWVA, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.426163245 (640h36m33.29724629s) [noise=true , address=145.79.227.41:4085 , piece_id=SBK75752ZTAHALZHLMUSFY6LQ5CFX7M6XE7USIFWQQCWDGKUCO7Q, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.426498062 (640h36m33.297581097s) [noise=true , address=77.90.31.13:4629 , piece_id=B3VBW3KAAHNOFEAUALRSFT4NUYDDR335IEZKDYHNHQRAPT4INEXQ, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.426856224 (640h36m33.29793927s) [noise=true , address=77.90.31.19:4842 , piece_id=L2AOC5E63OF4HGKR7CKCXSPU2OKNBICYNN2MDD4LRNYGNLKVSN4Q, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.427460885 (640h36m33.298543931s) [noise=true , address=145.79.227.9:4367 , piece_id=VF572O4NUNL6V3EVSHMY2FMFTYGFO66BG766NGX34PUE5D6IT5ZQ, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.427862600 (640h36m33.298945645s) [noise=true , address=77.90.31.12:4589 , piece_id=37BBHOW73QYQJDFGK36WPP733ZXTHUXKDSHJGT2JLXXAQEWCALPA, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.428023056 (640h36m33.299106091s) [noise=true , address=45.91.116.7:4009 , piece_id=GJGADK44KRYQRG773AQSBF67BCBBBECTSSHORUYICXQR6SD5E3KQ, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.428268934 (640h36m33.299351979s) [noise=true , address=45.91.116.4:4940 , piece_id=ETQF4GV3O5RCGFINVNORRXBISEKHLSGTKNQLC4O2AHZZKCYOKTHQ, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.428431534 (640h36m33.299514579s) [noise=true , address=145.79.227.4:4187 , piece_id=MONCIHND5Y7OMAPIERB7WX7TVX7NQX6T4OQF54LUPTIF37HHAJOQ, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.428636264 (640h36m33.299719309s) [noise=true , address=45.91.116.6:4975 , piece_id=LR4C5PTPP6B3KUIOEOOGQYNQZOVFUGA4ISRZSXKX6ZAU2PAFQJAQ, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.428918822 (640h36m33.300001867s) [noise=true , address=45.91.116.10:4084 , piece_id=XNOGQNIKCD7PNGHBBKM7CTKXMJ2XKAF3GQ75JUEENK6F6FCS2TOQ, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.429007680 (640h36m33.300090725s) [noise=true , address=77.90.31.16:4735 , piece_id=T2XYKIP6TWORE2OOHCBNMBFS3RGDOCVENFOPFHIMRMGN34IZ7GWA, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.429386582 (640h36m33.300469617s) [noise=true , address=77.90.31.11:4553 , piece_id=M7V5WCD4FYNJLQIQCZHB4F3LGTPNIQDWVM7XH5X7YKL6RZHVWCCA, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:35.136600720 (640h36m35.007683765s) [upload=561319]: Next returned nil segment
Jan 1 08:05:35.136614055 (640h36m35.00769709s) [upload=561319]: Waiting for error group managing segments...
Jan 1 08:05:35.418160256 (640h36m35.289243301s) [noise=true , address=145.79.227.26:4653 , piece_id=FPTDZAOGL56VZOYAVT2RF5RJSRDLAWEQ5WDBERCLPCDIAGPWJSXA, seg_pos={0 0}, upload=561319]: Done uploading piece. err: <nil>
Jan 1 08:05:35.434506661 (640h36m35.305589696s) [noise=true , address=145.79.227.9:4367 , piece_id=VF572O4NUNL6V3EVSHMY2FMFTYGFO66BG766NGX34PUE5D6IT5ZQ, seg_pos={0 0}, upload=561319]: Done uploading piece. err: <nil>
Jan 1 08:05:35.445590456 (640h36m35.316673501s) [noise=true , address=145.79.227.15:4873 , piece_id=DA6OMDRTENINDEKH7U7IA2FQ5P25OX7C2U6P5XC2VISHTIPQAWVA, seg_pos={0 0}, upload=561319]: Done uploading piece. err: <nil>
Jan 1 08:05:35.456856528 (640h36m35.327939563s) [noise=true , address=145.79.227.4:4187 , piece_id=MONCIHND5Y7OMAPIERB7WX7TVX7NQX6T4OQF54LUPTIF37HHAJOQ, seg_pos={0 0}, upload=561319]: Done uploading piece. err: <nil>
Jan 1 08:05:35.501865411 (640h36m35.372948446s) [noise=true , address=145.79.227.41:4085 , piece_id=SBK75752ZTAHALZHLMUSFY6LQ5CFX7M6XE7USIFWQQCWDGKUCO7Q, seg_pos={0 0}, upload=561319]: Done uploading piece. err: <nil>
Jan 1 08:05:35.507303396 (640h36m35.378386431s) [noise=true , address=145.79.227.3:4150 , piece_id=RKEABVHWH3CUB7EJG4U3TA6RPQ6M6VRGJQNSME2BPE2FX433CRSQ, seg_pos={0 0}, upload=561319]: Done uploading piece. err: <nil>
Jan 1 08:05:35.602364924 (640h36m35.473447959s) [noise=true , address=77.90.31.19:4842 , piece_id=L2AOC5E63OF4HGKR7CKCXSPU2OKNBICYNN2MDD4LRNYGNLKVSN4Q, seg_pos={0 0}, upload=561319]: Done uploading piece. err: <nil>
Jan 1 08:05:35.609000360 (640h36m35.480083405s) [noise=true , address=77.90.31.13:4629 , piece_id=B3VBW3KAAHNOFEAUALRSFT4NUYDDR335IEZKDYHNHQRAPT4INEXQ, seg_pos={0 0}, upload=561319]: Done uploading piece. err: <nil>
Jan 1 08:05:35.709299663 (640h36m35.580382698s) [noise=true , address=77.90.31.16:4735 , piece_id=T2XYKIP6TWORE2OOHCBNMBFS3RGDOCVENFOPFHIMRMGN34IZ7GWA, seg_pos={0 0}, upload=561319]: Done uploading piece. err: <nil>
Jan 1 08:05:35.711602518 (640h36m35.582685563s) [noise=true , address=77.90.31.12:4589 , piece_id=37BBHOW73QYQJDFGK36WPP733ZXTHUXKDSHJGT2JLXXAQEWCALPA, seg_pos={0 0}, upload=561319]: Done uploading piece. err: <nil>
Jan 1 08:05:35.720153349 (640h36m35.591236384s) [noise=true , address=45.91.116.6:4975 , piece_id=LR4C5PTPP6B3KUIOEOOGQYNQZOVFUGA4ISRZSXKX6ZAU2PAFQJAQ, seg_pos={0 0}, upload=561319]: Done uploading piece. err: <nil>
Jan 1 08:05:35.735457540 (640h36m35.606540576s) [noise=true , address=77.90.31.11:4553 , piece_id=M7V5WCD4FYNJLQIQCZHB4F3LGTPNIQDWVM7XH5X7YKL6RZHVWCCA, seg_pos={0 0}, upload=561319]: Done uploading piece. err: <nil>
Jan 1 08:05:35.850795468 (640h36m35.721878504s) [noise=true , address=45.91.116.4:4940 , piece_id=ETQF4GV3O5RCGFINVNORRXBISEKHLSGTKNQLC4O2AHZZKCYOKTHQ, seg_pos={0 0}, upload=561319]: Done uploading piece. err: <nil>
Jan 1 08:05:35.850814644 (640h36m35.72189769s) [seg_pos={0 0}, upload=561319]: Segment reached optimal threshold of 13 pieces.
Jan 1 08:05:35.850994106 (640h36m35.722077151s) [noise=true , address=45.91.116.10:4084 , piece_id=XNOGQNIKCD7PNGHBBKM7CTKXMJ2XKAF3GQ75JUEENK6F6FCS2TOQ, seg_pos={0 0}, upload=561319]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF
Jan 1 08:05:35.851000668 (640h36m35.722083713s) [noise=true , address=45.91.116.10:4084 , piece_id=XNOGQNIKCD7PNGHBBKM7CTKXMJ2XKAF3GQ75JUEENK6F6FCS2TOQ, seg_pos={0 0}, upload=561319]: Long tail cancellation due to optimal threshold
Jan 1 08:05:35.851128392 (640h36m35.722211427s) [noise=true , address=45.91.116.7:4009 , piece_id=GJGADK44KRYQRG773AQSBF67BCBBBECTSSHORUYICXQR6SD5E3KQ, seg_pos={0 0}, upload=561319]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Jan 1 08:05:35.851133912 (640h36m35.722216957s) [noise=true , address=45.91.116.7:4009 , piece_id=GJGADK44KRYQRG773AQSBF67BCBBBECTSSHORUYICXQR6SD5E3KQ, seg_pos={0 0}, upload=561319]: Long tail cancellation due to optimal threshold
Jan 1 08:05:35.851382736 (640h36m35.722465781s) [noise=true , address=45.91.116.8:4048 , piece_id=TKRTWZRZ5GTOPZBSNYKPODKEGZC3HG5HMRCSCXC56XY6UPR7ICSA, seg_pos={0 0}, upload=561319]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:36492->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:36492->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36492->45.91.116.8:4048: use of closed network connection
Jan 1 08:05:35.851402664 (640h36m35.722485709s) [noise=true , address=45.91.116.8:4048 , piece_id=TKRTWZRZ5GTOPZBSNYKPODKEGZC3HG5HMRCSCXC56XY6UPR7ICSA, seg_pos={0 0}, upload=561319]: Long tail cancellation due to optimal threshold
Jan 1 08:05:35.851460284 (640h36m35.722543329s) [seg_pos={0 0}, upload=561319]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 1 08:05:35.851566416 (640h36m35.722649471s) [upload=561319]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 1 08:05:35.851573239 (640h36m35.722656274s) [upload=561319]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 1 08:05:36.144226899 (640h36m36.015309944s) [upload=561319]: Done uploading segments...
Jan 1 08:05:36.144246647 (640h36m36.015329692s) [upload=561319]: Upload finished. err: <nil>
Duration: 5.934203583s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................X......................] duration:
8.357204347s trace:
https://jaeger.dev.storj.io/trace/44e9d74e3b154e8e
FAIL (2026-01-01 08:06:31 UTC)
orbiter warning: context deadline exceeded
Duration: 8.357235457s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................X.....................] duration:
16.697183361s trace:
https://jaeger.dev.storj.io/trace/44e9d74e3b154e8e
FAIL (2026-01-01 08:06:14 UTC)
orbiter warning: context deadline exceeded
Duration: 16.697204211s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [............X.....X..XX.........................XX..........] duration:
29.604745567s trace:
https://jaeger.dev.storj.io/trace/10d96081b66cd020
FAIL (2026-01-01 08:06:13 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
73329s) [noise=true , address=145.79.227.40:4066 , piece_id=ABZ7356MEBO4XPYAKFSS4EQJYJN4CWNJR3C6NXFWLAEO65KL3J5Q, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.722330803 (640h29m1.593413848s) [noise=true , address=145.79.227.9:4391 , piece_id=XS6HYQTJO355YLXEIDBKHTJ44UAT5UBKHDUCFA744X6TBPGYN5YQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.722848709 (640h29m1.593931754s) [noise=true , address=77.90.31.12:4606 , piece_id=K4E5ECNMOMKEIFHZ3EZ2QSQHZ5IYP3CAD5DA6DBQLM4JYMOYU7MA, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.723452529 (640h29m1.594535574s) [noise=true , address=193.149.164.143:4694 , piece_id=EUO4UWMROIFFAJECCFRJBQF4XV25VDKGKX3UF2PTUMTE5JBGD67Q, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.723730728 (640h29m1.594813773s) [noise=true , address=45.91.116.5:4320 , piece_id=FOFLP7277VUUPTRZZHQLNVJ3YVYBELJTUYL7PTT7FJNN7J7QLS2A, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.724307808 (640h29m1.595390853s) [noise=true , address=64.49.13.169:4705 , piece_id=2P3JANSNK64726X7IYJOIQOKAQ4P4LJ5A2GVPX6UXOWUDIYRPZMA, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.724847144 (640h29m1.595930189s) [noise=true , address=64.49.13.137:4494 , piece_id=GCFOH6XRKDEGNG6TFPCMHG5RUCP2LUFS3XEOPNF42VYTGS63ZHQQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.724970960 (640h29m1.596054005s) [noise=true , address=77.90.31.15:4725 , piece_id=LLGZBPL7CHDI2TAY2SYMUMC264USSUQRVFR5P4HTLBWPBOTSUSOA, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.725289036 (640h29m1.596372071s) [noise=true , address=145.79.227.49:4371 , piece_id=RGZZMDFFVVSIQJ2ZEUSK275SBWMGV75WGNDLY6LJ2TNRPUSY7RMQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.725518012 (640h29m1.596601047s) [noise=true , address=77.90.31.13:4649 , piece_id=YRKZDAT5O2USQM4SO5GRZ45SX7MA3U52KN2CRGYPX5WJ4HIKOJBA, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.725902264 (640h29m1.596985309s) [noise=true , address=45.91.116.12:4238 , piece_id=R5LPEYTYC3XB76SSMDVHPZORAWB2P3S4TETMYV23NJI4UXYUYCEQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.726176987 (640h29m1.597260032s) [noise=true , address=145.79.227.35:4497 , piece_id=YDNRFAJVXHRJD5QHICHR6CGF6K5ARZTLHOUSRLV6JAYU3ZSWEY3A, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.726843697 (640h29m1.597926742s) [noise=true , address=77.90.31.19:4846 , piece_id=2XJSN2WTR2BAVKDJZXEMY6SRX7KLOU4SVGYH7TNJIJTPW723DEHQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.727470170 (640h29m1.598553225s) [noise=true , address=145.79.227.30:4784 , piece_id=HA6Z6M3HASDLIDP2PHS7MUDOHBF6T6ZNPTABXF5PYXR2HFYSBOPQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.728088075 (640h29m1.59917112s) [noise=true , address=145.79.227.39:4032 , piece_id=Z7BQPDXOVQ6AEKACBB2DJFI652AT6UIHXUB2HPUJFMK53EF73AWQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:03.152355605 (640h29m3.02343864s) [upload=561203]: Next returned nil segment
Jan 1 07:58:03.152371765 (640h29m3.023454811s) [upload=561203]: Waiting for error group managing segments...
Jan 1 07:58:03.563991563 (640h29m3.435074608s) [noise=true , address=145.79.227.9:4391 , piece_id=XS6HYQTJO355YLXEIDBKHTJ44UAT5UBKHDUCFA744X6TBPGYN5YQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:03.652285026 (640h29m3.523368071s) [noise=true , address=145.79.227.49:4371 , piece_id=RGZZMDFFVVSIQJ2ZEUSK275SBWMGV75WGNDLY6LJ2TNRPUSY7RMQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:03.671359306 (640h29m3.542442351s) [noise=true , address=145.79.227.39:4032 , piece_id=Z7BQPDXOVQ6AEKACBB2DJFI652AT6UIHXUB2HPUJFMK53EF73AWQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:03.774991565 (640h29m3.64607462s) [noise=true , address=77.90.31.19:4846 , piece_id=2XJSN2WTR2BAVKDJZXEMY6SRX7KLOU4SVGYH7TNJIJTPW723DEHQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:03.870781221 (640h29m3.741864266s) [noise=true , address=77.90.31.12:4606 , piece_id=K4E5ECNMOMKEIFHZ3EZ2QSQHZ5IYP3CAD5DA6DBQLM4JYMOYU7MA, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:03.878179488 (640h29m3.749262533s) [noise=true , address=77.90.31.13:4649 , piece_id=YRKZDAT5O2USQM4SO5GRZ45SX7MA3U52KN2CRGYPX5WJ4HIKOJBA, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:03.977331467 (640h29m3.848414512s) [noise=true , address=77.90.31.15:4725 , piece_id=LLGZBPL7CHDI2TAY2SYMUMC264USSUQRVFR5P4HTLBWPBOTSUSOA, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.224768738 (640h29m4.095851783s) [noise=true , address=45.91.116.12:4238 , piece_id=R5LPEYTYC3XB76SSMDVHPZORAWB2P3S4TETMYV23NJI4UXYUYCEQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.457462948 (640h29m4.328545994s) [noise=true , address=45.91.116.5:4320 , piece_id=FOFLP7277VUUPTRZZHQLNVJ3YVYBELJTUYL7PTT7FJNN7J7QLS2A, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.488291199 (640h29m4.359374234s) [noise=true , address=145.79.227.35:4497 , piece_id=YDNRFAJVXHRJD5QHICHR6CGF6K5ARZTLHOUSRLV6JAYU3ZSWEY3A, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.522541837 (640h29m4.393624882s) [noise=true , address=145.79.227.43:4171 , piece_id=KYGE2RFI6O6DUFWVGJV2IT5VE3U43WJ4ODNQMTZFQH2DAADFC6BQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.525844266 (640h29m4.396927321s) [noise=true , address=145.79.227.33:4886 , piece_id=CYGCX4TM2ZFPCJYVDF52PDNR2OCNZ2NUSUUOUMUHYGCADWWFDHVQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.539045983 (640h29m4.410129028s) [noise=true , address=145.79.227.34:4924 , piece_id=QUPSGORFPMQVD2IRAK7COO2EE7XDV2KWEBTI2FQ2C25RTAAFNSRA, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.544558039 (640h29m4.415641084s) [noise=true , address=145.79.227.30:4784 , piece_id=HA6Z6M3HASDLIDP2PHS7MUDOHBF6T6ZNPTABXF5PYXR2HFYSBOPQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.544582956 (640h29m4.415665991s) [seg_pos={0 0}, upload=561203]: Segment reached optimal threshold of 14 pieces.
Jan 1 07:58:04.544811462 (640h29m4.415894527s) [noise=true , address=193.149.164.143:4694 , piece_id=EUO4UWMROIFFAJECCFRJBQF4XV25VDKGKX3UF2PTUMTE5JBGD67Q, seg_pos={0 0}, upload=561203]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx): protocol: expected piece hash; context canceled; EOF
Jan 1 07:58:04.544821240 (640h29m4.415904275s) [noise=true , address=193.149.164.143:4694 , piece_id=EUO4UWMROIFFAJECCFRJBQF4XV25VDKGKX3UF2PTUMTE5JBGD67Q, seg_pos={0 0}, upload=561203]: Long tail cancellation due to optimal threshold
Jan 1 07:58:04.545116732 (640h29m4.416199767s) [noise=true , address=64.49.13.169:4705 , piece_id=2P3JANSNK64726X7IYJOIQOKAQ4P4LJ5A2GVPX6UXOWUDIYRPZMA, seg_pos={0 0}, upload=561203]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:45312->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:45312->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45312->64.49.13.169:4705: use of closed network connection
Jan 1 07:58:04.545123666 (640h29m4.416206711s) [noise=true , address=64.49.13.169:4705 , piece_id=2P3JANSNK64726X7IYJOIQOKAQ4P4LJ5A2GVPX6UXOWUDIYRPZMA, seg_pos={0 0}, upload=561203]: Long tail cancellation due to optimal threshold
Jan 1 07:58:04.545303088 (640h29m4.416386123s) [noise=true , address=145.79.227.40:4066 , piece_id=ABZ7356MEBO4XPYAKFSS4EQJYJN4CWNJR3C6NXFWLAEO65KL3J5Q, seg_pos={0 0}, upload=561203]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF
Jan 1 07:58:04.545312055 (640h29m4.4163951s) [noise=true , address=145.79.227.40:4066 , piece_id=ABZ7356MEBO4XPYAKFSS4EQJYJN4CWNJR3C6NXFWLAEO65KL3J5Q, seg_pos={0 0}, upload=561203]: Long tail cancellation due to optimal threshold
Jan 1 07:58:04.545414279 (640h29m4.416497314s) [noise=true , address=64.49.13.137:4494 , piece_id=GCFOH6XRKDEGNG6TFPCMHG5RUCP2LUFS3XEOPNF42VYTGS63ZHQQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TX66fDVv3qu4yBGx9FNZTiArNeUnqxjMYwb3VGqYCQRe8RwoU): protocol: write tcp 188.166.225.170:35392->64.49.13.137:4494: use of closed network connection; write tcp 188.166.225.170:35392->64.49.13.137:4494: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35392->64.49.13.137:4494: use of closed network connection
Jan 1 07:58:04.545422815 (640h29m4.41650585s) [noise=true , address=64.49.13.137:4494 , piece_id=GCFOH6XRKDEGNG6TFPCMHG5RUCP2LUFS3XEOPNF42VYTGS63ZHQQ, seg_pos={0 0}, upload=561203]: Long tail cancellation due to optimal threshold
Jan 1 07:58:04.545439939 (640h29m4.416522974s) [seg_pos={0 0}, upload=561203]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 1 07:58:04.545580406 (640h29m4.416663441s) [upload=561203]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 1 07:58:04.545587800 (640h29m4.416670845s) [upload=561203]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 1 07:58:04.840726371 (640h29m4.711809416s) [upload=561203]: Done uploading segments...
Jan 1 07:58:04.840749646 (640h29m4.711832691s) [upload=561203]: Upload finished. err: <nil>
Duration: 29.604776286s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
20 out of 60 is failed (threshold=3) [.XX.XX..X...X.XX.X...X.........X.X........XX.X...X..X....XXX] duration:
30.003226493s trace:
https://jaeger.dev.storj.io/trace/20c951f57c07dfa9
FAIL (2026-01-01 08:06:02 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: B3VBW3KAAHNOFEAUALRSFT4NUYDDR335IEZKDYHNHQRAPT4INEXQ) context deadline exceeded; piecestore: (Node ID: 1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv, Piece ID: 37BBHOW73QYQJDFGK36WPP733ZXTHUXKDSHJGT2JLXXAQEWCALPA) context deadline exceeded
======LOGS======
Jan 1 08:05:33.167157584 (640h36m33.038240619s) [upload=561319]: Starting upload
Jan 1 08:05:33.167395737 (640h36m33.038478782s) [upload=561319]: Done starting upload
Jan 1 08:05:33.167588645 (640h36m33.03867169s) [upload=561319]: Uploading segments...
Jan 1 08:05:33.167794657 (640h36m33.038877693s) [upload=561319]: Got next segment. Inline: false
Jan 1 08:05:33.167935155 (640h36m33.0390182s) [upload=561319]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 1 08:05:33.167959080 (640h36m33.039042115s) [upload=561319]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 1 08:05:33.425006192 (640h36m33.296089237s) [seg_pos={0 0}, upload=561319]: Begin upload segment...
Jan 1 08:05:33.425075253 (640h36m33.296158298s) [seg_pos={0 0}, upload=561319]: Done begin upload segment.
Jan 1 08:05:33.425092566 (640h36m33.296175601s) [noise=true , address=145.79.227.3:4150 , piece_id=RKEABVHWH3CUB7EJG4U3TA6RPQ6M6VRGJQNSME2BPE2FX433CRSQ, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.425116662 (640h36m33.296199707s) [noise=true , address=145.79.227.26:4653 , piece_id=FPTDZAOGL56VZOYAVT2RF5RJSRDLAWEQ5WDBERCLPCDIAGPWJSXA, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.425560297 (640h36m33.296643332s) [noise=true , address=45.91.116.8:4048 , piece_id=TKRTWZRZ5GTOPZBSNYKPODKEGZC3HG5HMRCSCXC56XY6UPR7ICSA, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.425818839 (640h36m33.296901884s) [noise=true , address=145.79.227.15:4873 , piece_id=DA6OMDRTENINDEKH7U7IA2FQ5P25OX7C2U6P5XC2VISHTIPQAWVA, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.426163245 (640h36m33.29724629s) [noise=true , address=145.79.227.41:4085 , piece_id=SBK75752ZTAHALZHLMUSFY6LQ5CFX7M6XE7USIFWQQCWDGKUCO7Q, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.426498062 (640h36m33.297581097s) [noise=true , address=77.90.31.13:4629 , piece_id=B3VBW3KAAHNOFEAUALRSFT4NUYDDR335IEZKDYHNHQRAPT4INEXQ, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.426856224 (640h36m33.29793927s) [noise=true , address=77.90.31.19:4842 , piece_id=L2AOC5E63OF4HGKR7CKCXSPU2OKNBICYNN2MDD4LRNYGNLKVSN4Q, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.427460885 (640h36m33.298543931s) [noise=true , address=145.79.227.9:4367 , piece_id=VF572O4NUNL6V3EVSHMY2FMFTYGFO66BG766NGX34PUE5D6IT5ZQ, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.427862600 (640h36m33.298945645s) [noise=true , address=77.90.31.12:4589 , piece_id=37BBHOW73QYQJDFGK36WPP733ZXTHUXKDSHJGT2JLXXAQEWCALPA, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.428023056 (640h36m33.299106091s) [noise=true , address=45.91.116.7:4009 , piece_id=GJGADK44KRYQRG773AQSBF67BCBBBECTSSHORUYICXQR6SD5E3KQ, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.428268934 (640h36m33.299351979s) [noise=true , address=45.91.116.4:4940 , piece_id=ETQF4GV3O5RCGFINVNORRXBISEKHLSGTKNQLC4O2AHZZKCYOKTHQ, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.428431534 (640h36m33.299514579s) [noise=true , address=145.79.227.4:4187 , piece_id=MONCIHND5Y7OMAPIERB7WX7TVX7NQX6T4OQF54LUPTIF37HHAJOQ, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.428636264 (640h36m33.299719309s) [noise=true , address=45.91.116.6:4975 , piece_id=LR4C5PTPP6B3KUIOEOOGQYNQZOVFUGA4ISRZSXKX6ZAU2PAFQJAQ, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.428918822 (640h36m33.300001867s) [noise=true , address=45.91.116.10:4084 , piece_id=XNOGQNIKCD7PNGHBBKM7CTKXMJ2XKAF3GQ75JUEENK6F6FCS2TOQ, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.429007680 (640h36m33.300090725s) [noise=true , address=77.90.31.16:4735 , piece_id=T2XYKIP6TWORE2OOHCBNMBFS3RGDOCVENFOPFHIMRMGN34IZ7GWA, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:33.429386582 (640h36m33.300469617s) [noise=true , address=77.90.31.11:4553 , piece_id=M7V5WCD4FYNJLQIQCZHB4F3LGTPNIQDWVM7XH5X7YKL6RZHVWCCA, seg_pos={0 0}, upload=561319]: Uploading piece...
Jan 1 08:05:35.136600720 (640h36m35.007683765s) [upload=561319]: Next returned nil segment
Jan 1 08:05:35.136614055 (640h36m35.00769709s) [upload=561319]: Waiting for error group managing segments...
Jan 1 08:05:35.418160256 (640h36m35.289243301s) [noise=true , address=145.79.227.26:4653 , piece_id=FPTDZAOGL56VZOYAVT2RF5RJSRDLAWEQ5WDBERCLPCDIAGPWJSXA, seg_pos={0 0}, upload=561319]: Done uploading piece. err: <nil>
Jan 1 08:05:35.434506661 (640h36m35.305589696s) [noise=true , address=145.79.227.9:4367 , piece_id=VF572O4NUNL6V3EVSHMY2FMFTYGFO66BG766NGX34PUE5D6IT5ZQ, seg_pos={0 0}, upload=561319]: Done uploading piece. err: <nil>
Jan 1 08:05:35.445590456 (640h36m35.316673501s) [noise=true , address=145.79.227.15:4873 , piece_id=DA6OMDRTENINDEKH7U7IA2FQ5P25OX7C2U6P5XC2VISHTIPQAWVA, seg_pos={0 0}, upload=561319]: Done uploading piece. err: <nil>
Jan 1 08:05:35.456856528 (640h36m35.327939563s) [noise=true , address=145.79.227.4:4187 , piece_id=MONCIHND5Y7OMAPIERB7WX7TVX7NQX6T4OQF54LUPTIF37HHAJOQ, seg_pos={0 0}, upload=561319]: Done uploading piece. err: <nil>
Jan 1 08:05:35.501865411 (640h36m35.372948446s) [noise=true , address=145.79.227.41:4085 , piece_id=SBK75752ZTAHALZHLMUSFY6LQ5CFX7M6XE7USIFWQQCWDGKUCO7Q, seg_pos={0 0}, upload=561319]: Done uploading piece. err: <nil>
Jan 1 08:05:35.507303396 (640h36m35.378386431s) [noise=true , address=145.79.227.3:4150 , piece_id=RKEABVHWH3CUB7EJG4U3TA6RPQ6M6VRGJQNSME2BPE2FX433CRSQ, seg_pos={0 0}, upload=561319]: Done uploading piece. err: <nil>
Jan 1 08:05:35.602364924 (640h36m35.473447959s) [noise=true , address=77.90.31.19:4842 , piece_id=L2AOC5E63OF4HGKR7CKCXSPU2OKNBICYNN2MDD4LRNYGNLKVSN4Q, seg_pos={0 0}, upload=561319]: Done uploading piece. err: <nil>
Jan 1 08:05:35.609000360 (640h36m35.480083405s) [noise=true , address=77.90.31.13:4629 , piece_id=B3VBW3KAAHNOFEAUALRSFT4NUYDDR335IEZKDYHNHQRAPT4INEXQ, seg_pos={0 0}, upload=561319]: Done uploading piece. err: <nil>
Jan 1 08:05:35.709299663 (640h36m35.580382698s) [noise=true , address=77.90.31.16:4735 , piece_id=T2XYKIP6TWORE2OOHCBNMBFS3RGDOCVENFOPFHIMRMGN34IZ7GWA, seg_pos={0 0}, upload=561319]: Done uploading piece. err: <nil>
Jan 1 08:05:35.711602518 (640h36m35.582685563s) [noise=true , address=77.90.31.12:4589 , piece_id=37BBHOW73QYQJDFGK36WPP733ZXTHUXKDSHJGT2JLXXAQEWCALPA, seg_pos={0 0}, upload=561319]: Done uploading piece. err: <nil>
Jan 1 08:05:35.720153349 (640h36m35.591236384s) [noise=true , address=45.91.116.6:4975 , piece_id=LR4C5PTPP6B3KUIOEOOGQYNQZOVFUGA4ISRZSXKX6ZAU2PAFQJAQ, seg_pos={0 0}, upload=561319]: Done uploading piece. err: <nil>
Jan 1 08:05:35.735457540 (640h36m35.606540576s) [noise=true , address=77.90.31.11:4553 , piece_id=M7V5WCD4FYNJLQIQCZHB4F3LGTPNIQDWVM7XH5X7YKL6RZHVWCCA, seg_pos={0 0}, upload=561319]: Done uploading piece. err: <nil>
Jan 1 08:05:35.850795468 (640h36m35.721878504s) [noise=true , address=45.91.116.4:4940 , piece_id=ETQF4GV3O5RCGFINVNORRXBISEKHLSGTKNQLC4O2AHZZKCYOKTHQ, seg_pos={0 0}, upload=561319]: Done uploading piece. err: <nil>
Jan 1 08:05:35.850814644 (640h36m35.72189769s) [seg_pos={0 0}, upload=561319]: Segment reached optimal threshold of 13 pieces.
Jan 1 08:05:35.850994106 (640h36m35.722077151s) [noise=true , address=45.91.116.10:4084 , piece_id=XNOGQNIKCD7PNGHBBKM7CTKXMJ2XKAF3GQ75JUEENK6F6FCS2TOQ, seg_pos={0 0}, upload=561319]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: expected piece hash; context canceled; EOF
Jan 1 08:05:35.851000668 (640h36m35.722083713s) [noise=true , address=45.91.116.10:4084 , piece_id=XNOGQNIKCD7PNGHBBKM7CTKXMJ2XKAF3GQ75JUEENK6F6FCS2TOQ, seg_pos={0 0}, upload=561319]: Long tail cancellation due to optimal threshold
Jan 1 08:05:35.851128392 (640h36m35.722211427s) [noise=true , address=45.91.116.7:4009 , piece_id=GJGADK44KRYQRG773AQSBF67BCBBBECTSSHORUYICXQR6SD5E3KQ, seg_pos={0 0}, upload=561319]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Jan 1 08:05:35.851133912 (640h36m35.722216957s) [noise=true , address=45.91.116.7:4009 , piece_id=GJGADK44KRYQRG773AQSBF67BCBBBECTSSHORUYICXQR6SD5E3KQ, seg_pos={0 0}, upload=561319]: Long tail cancellation due to optimal threshold
Jan 1 08:05:35.851382736 (640h36m35.722465781s) [noise=true , address=45.91.116.8:4048 , piece_id=TKRTWZRZ5GTOPZBSNYKPODKEGZC3HG5HMRCSCXC56XY6UPR7ICSA, seg_pos={0 0}, upload=561319]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: write tcp 188.166.225.170:36492->45.91.116.8:4048: use of closed network connection; write tcp 188.166.225.170:36492->45.91.116.8:4048: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36492->45.91.116.8:4048: use of closed network connection
Jan 1 08:05:35.851402664 (640h36m35.722485709s) [noise=true , address=45.91.116.8:4048 , piece_id=TKRTWZRZ5GTOPZBSNYKPODKEGZC3HG5HMRCSCXC56XY6UPR7ICSA, seg_pos={0 0}, upload=561319]: Long tail cancellation due to optimal threshold
Jan 1 08:05:35.851460284 (640h36m35.722543329s) [seg_pos={0 0}, upload=561319]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 1 08:05:35.851566416 (640h36m35.722649471s) [upload=561319]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 1 08:05:35.851573239 (640h36m35.722656274s) [upload=561319]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 1 08:05:36.144226899 (640h36m36.015309944s) [upload=561319]: Done uploading segments...
Jan 1 08:05:36.144246647 (640h36m36.015329692s) [upload=561319]: Upload finished. err: <nil>
Duration: 30.003252331s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................X....................] duration:
20.271023626s trace:
https://jaeger.dev.storj.io/trace/44e9d74e3b154e8e
FAIL (2026-01-01 08:05:56 UTC)
orbiter warning: context deadline exceeded
Duration: 20.27104092s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
39 out of 60 is failed (threshold=6) [XX.X.XXXXX.XX..XXXXX.XXXX..XX.XXX.XX...X.X..X...XXXXXX..XXXX] duration:
5.001962062s trace:
https://jaeger.dev.storj.io/trace/5cbad9b34042f0de
FAIL (2026-01-01 08:05:50 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001984735s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................X...................] duration:
21.69145605s trace:
https://jaeger.dev.storj.io/trace/44e9d74e3b154e8e
FAIL (2026-01-01 08:05:31 UTC)
orbiter warning: context deadline exceeded
Duration: 21.691474858s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
20 out of 60 is failed (threshold=3) [X.XX.XX..X...X.XX.X...X.........X.X........XX.X...X..X....XX] duration:
30.003314454s trace:
https://jaeger.dev.storj.io/trace/1a89b0d059c01ae5
FAIL (2026-01-01 08:05:25 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
56.208992724 (640h35m56.080075759s) [upload=561312]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 1 08:04:56.457246973 (640h35m56.328330028s) [seg_pos={0 0}, upload=561312]: Begin upload segment...
Jan 1 08:04:56.457309221 (640h35m56.328392266s) [seg_pos={0 0}, upload=561312]: Done begin upload segment.
Jan 1 08:04:56.457339920 (640h35m56.328422965s) [noise=true , address=145.79.227.22:4510 , piece_id=W2ERBY5SAN3LN4GN7GL24T27PRBW2IAHHMOPUHXWVE3SF3YLE3DQ, seg_pos={0 0}, upload=561312]: Uploading piece...
Jan 1 08:04:56.457349177 (640h35m56.328432222s) [noise=true , address=145.79.227.13:4804 , piece_id=TTTXSBR3PFGZRNJIHBXILUCW7NQAMOBLFBCCKI6X3KKUKGVD4BDQ, seg_pos={0 0}, upload=561312]: Uploading piece...
Jan 1 08:04:56.457714041 (640h35m56.328797097s) [noise=true , address=145.79.227.4:4185 , piece_id=IHJQF475ORAADGCTDFLVJNSAAAXSFG43QNMV5D2Y4IUK23BYCXCQ, seg_pos={0 0}, upload=561312]: Uploading piece...
Jan 1 08:04:56.459106303 (640h35m56.330189338s) [noise=true , address=145.79.227.19:4078 , piece_id=M56C73K4A2GIHFNZWMZMPD2LURDSV7G7LM7QL3OKG6LXXRRYIWLQ, seg_pos={0 0}, upload=561312]: Uploading piece...
Jan 1 08:04:56.459546982 (640h35m56.330630027s) [noise=true , address=145.79.227.26:4653 , piece_id=EVL5QYWAXLHIW3V7BCO32EYVISCUN2AJ4UUMV57X3ML4ZGOZXWQQ, seg_pos={0 0}, upload=561312]: Uploading piece...
Jan 1 08:04:56.459973895 (640h35m56.33105694s) [noise=true , address=77.90.31.14:4664 , piece_id=CAGDEGYB2537JB3VPBX5TSWT3Q6NF66TVBETJGGWJERJRH3L5A2A, seg_pos={0 0}, upload=561312]: Uploading piece...
Jan 1 08:04:56.459982832 (640h35m56.331065878s) [noise=true , address=45.91.116.8:4047 , piece_id=S65MMQ7U2R2V4XVC52XN2UF5XQLYF4UODI5QQ23HNSYAQ4YJK4FA, seg_pos={0 0}, upload=561312]: Uploading piece...
Jan 1 08:04:56.460342727 (640h35m56.331425763s) [noise=true , address=145.79.227.31:4797 , piece_id=E3XQ56SF2OQTG3HUUMZ5M4AB4LGHJTX62DHXRBUOZVN6NGQIIQEQ, seg_pos={0 0}, upload=561312]: Uploading piece...
Jan 1 08:04:56.460385469 (640h35m56.331468514s) [noise=true , address=145.79.227.52:4447 , piece_id=FUUZ5LVPLNYKH5LE5XEUIPULLISELGJ27RQFHYCBMBPOTDKNS53A, seg_pos={0 0}, upload=561312]: Uploading piece...
Jan 1 08:04:56.460858820 (640h35m56.331941855s) [noise=true , address=45.91.116.11:4117 , piece_id=KSYAXDVZ6QFL3GBCYBHMVJZMBDDB4FC7KTAZEJKB3J2JEEAEN3ZA, seg_pos={0 0}, upload=561312]: Uploading piece...
Jan 1 08:04:56.461287466 (640h35m56.332370501s) [noise=true , address=145.79.227.17:4972 , piece_id=YQHJMWIHOMAPKIIOMCT63ZODN34GDWQBSLVLZ4X3PTMPJEFZ467A, seg_pos={0 0}, upload=561312]: Uploading piece...
Jan 1 08:04:56.461893942 (640h35m56.332976977s) [noise=true , address=45.91.116.7:4010 , piece_id=ZN74ZBDLV3JODYB7K7SP6K3RT3ZUQHAVOPVJS7JOAWJIJ655TF2Q, seg_pos={0 0}, upload=561312]: Uploading piece...
Jan 1 08:04:56.462253506 (640h35m56.333336601s) [noise=true , address=145.79.227.21:4475 , piece_id=DDLFHSB2LA7T2YG4DBUEABS2PB77FECNPBVN7LMVMM7AIG5RNMFA, seg_pos={0 0}, upload=561312]: Uploading piece...
Jan 1 08:04:56.462344378 (640h35m56.333427434s) [noise=true , address=145.79.227.41:4087 , piece_id=4ZWCRH2NJ4QB3OEYTTWPOLJLCJECIX7G2SNYGXRCLN7RUEDMGP2Q, seg_pos={0 0}, upload=561312]: Uploading piece...
Jan 1 08:04:56.462732148 (640h35m56.333815183s) [noise=true , address=77.90.31.21:4878 , piece_id=UCGAIABSEZ32P33INMFDQPW4AIUUCJMVY3ZFPRZTW5OBIJ6FBFJA, seg_pos={0 0}, upload=561312]: Uploading piece...
Jan 1 08:04:56.463083877 (640h35m56.334166912s) [noise=true , address=145.79.227.39:4014 , piece_id=H7W3ABZQU7VSRI2BWB74EKB4LHJWZRHDCK5QNK4K77VFYVFBPWNA, seg_pos={0 0}, upload=561312]: Uploading piece...
Jan 1 08:04:58.171524079 (640h35m58.042607114s) [upload=561312]: Next returned nil segment
Jan 1 08:04:58.171534598 (640h35m58.042617633s) [upload=561312]: Waiting for error group managing segments...
Jan 1 08:04:58.455914284 (640h35m58.326997329s) [noise=true , address=145.79.227.31:4797 , piece_id=E3XQ56SF2OQTG3HUUMZ5M4AB4LGHJTX62DHXRBUOZVN6NGQIIQEQ, seg_pos={0 0}, upload=561312]: Done uploading piece. err: <nil>
Jan 1 08:04:58.456249763 (640h35m58.327332808s) [noise=true , address=145.79.227.41:4087 , piece_id=4ZWCRH2NJ4QB3OEYTTWPOLJLCJECIX7G2SNYGXRCLN7RUEDMGP2Q, seg_pos={0 0}, upload=561312]: Done uploading piece. err: <nil>
Jan 1 08:04:58.472518440 (640h35m58.343601475s) [noise=true , address=145.79.227.39:4014 , piece_id=H7W3ABZQU7VSRI2BWB74EKB4LHJWZRHDCK5QNK4K77VFYVFBPWNA, seg_pos={0 0}, upload=561312]: Done uploading piece. err: <nil>
Jan 1 08:04:58.473622884 (640h35m58.344705919s) [noise=true , address=145.79.227.17:4972 , piece_id=YQHJMWIHOMAPKIIOMCT63ZODN34GDWQBSLVLZ4X3PTMPJEFZ467A, seg_pos={0 0}, upload=561312]: Done uploading piece. err: <nil>
Jan 1 08:04:58.475484428 (640h35m58.346567463s) [noise=true , address=145.79.227.22:4510 , piece_id=W2ERBY5SAN3LN4GN7GL24T27PRBW2IAHHMOPUHXWVE3SF3YLE3DQ, seg_pos={0 0}, upload=561312]: Done uploading piece. err: <nil>
Jan 1 08:04:58.479843499 (640h35m58.350926534s) [noise=true , address=145.79.227.52:4447 , piece_id=FUUZ5LVPLNYKH5LE5XEUIPULLISELGJ27RQFHYCBMBPOTDKNS53A, seg_pos={0 0}, upload=561312]: Done uploading piece. err: <nil>
Jan 1 08:04:58.485423603 (640h35m58.356506648s) [noise=true , address=145.79.227.21:4475 , piece_id=DDLFHSB2LA7T2YG4DBUEABS2PB77FECNPBVN7LMVMM7AIG5RNMFA, seg_pos={0 0}, upload=561312]: Done uploading piece. err: <nil>
Jan 1 08:04:58.486683283 (640h35m58.357766318s) [noise=true , address=145.79.227.19:4078 , piece_id=M56C73K4A2GIHFNZWMZMPD2LURDSV7G7LM7QL3OKG6LXXRRYIWLQ, seg_pos={0 0}, upload=561312]: Done uploading piece. err: <nil>
Jan 1 08:04:58.532637135 (640h35m58.40372017s) [noise=true , address=145.79.227.13:4804 , piece_id=TTTXSBR3PFGZRNJIHBXILUCW7NQAMOBLFBCCKI6X3KKUKGVD4BDQ, seg_pos={0 0}, upload=561312]: Done uploading piece. err: <nil>
Jan 1 08:04:58.590858355 (640h35m58.46194139s) [noise=true , address=145.79.227.4:4185 , piece_id=IHJQF475ORAADGCTDFLVJNSAAAXSFG43QNMV5D2Y4IUK23BYCXCQ, seg_pos={0 0}, upload=561312]: Done uploading piece. err: <nil>
Jan 1 08:04:58.748821207 (640h35m58.619904252s) [noise=true , address=145.79.227.26:4653 , piece_id=EVL5QYWAXLHIW3V7BCO32EYVISCUN2AJ4UUMV57X3ML4ZGOZXWQQ, seg_pos={0 0}, upload=561312]: Done uploading piece. err: <nil>
Jan 1 08:04:58.776834177 (640h35m58.647917212s) [noise=true , address=45.91.116.11:4117 , piece_id=KSYAXDVZ6QFL3GBCYBHMVJZMBDDB4FC7KTAZEJKB3J2JEEAEN3ZA, seg_pos={0 0}, upload=561312]: Done uploading piece. err: <nil>
Jan 1 08:04:58.838487964 (640h35m58.709570999s) [noise=true , address=45.91.116.7:4010 , piece_id=ZN74ZBDLV3JODYB7K7SP6K3RT3ZUQHAVOPVJS7JOAWJIJ655TF2Q, seg_pos={0 0}, upload=561312]: Done uploading piece. err: <nil>
Jan 1 08:04:58.838509384 (640h35m58.70959243s) [seg_pos={0 0}, upload=561312]: Segment reached optimal threshold of 13 pieces.
Jan 1 08:04:58.838778688 (640h35m58.709861733s) [noise=true , address=45.91.116.8:4047 , piece_id=S65MMQ7U2R2V4XVC52XN2UF5XQLYF4UODI5QQ23HNSYAQ4YJK4FA, seg_pos={0 0}, upload=561312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: write tcp 188.166.225.170:48678->45.91.116.8:4047: use of closed network connection; write tcp 188.166.225.170:48678->45.91.116.8:4047: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48678->45.91.116.8:4047: use of closed network connection
Jan 1 08:04:58.838789307 (640h35m58.709872343s) [noise=true , address=45.91.116.8:4047 , piece_id=S65MMQ7U2R2V4XVC52XN2UF5XQLYF4UODI5QQ23HNSYAQ4YJK4FA, seg_pos={0 0}, upload=561312]: Long tail cancellation due to optimal threshold
Jan 1 08:04:58.839002774 (640h35m58.710085819s) [noise=true , address=77.90.31.21:4878 , piece_id=UCGAIABSEZ32P33INMFDQPW4AIUUCJMVY3ZFPRZTW5OBIJ6FBFJA, seg_pos={0 0}, upload=561312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:33020->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:33020->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33020->77.90.31.21:4878: use of closed network connection
Jan 1 08:04:58.839056026 (640h35m58.710139061s) [noise=true , address=77.90.31.21:4878 , piece_id=UCGAIABSEZ32P33INMFDQPW4AIUUCJMVY3ZFPRZTW5OBIJ6FBFJA, seg_pos={0 0}, upload=561312]: Long tail cancellation due to optimal threshold
Jan 1 08:04:58.839152199 (640h35m58.710235244s) [noise=true , address=77.90.31.14:4664 , piece_id=CAGDEGYB2537JB3VPBX5TSWT3Q6NF66TVBETJGGWJERJRH3L5A2A, seg_pos={0 0}, upload=561312]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: write tcp 188.166.225.170:49346->77.90.31.14:4664: use of closed network connection; write tcp 188.166.225.170:49346->77.90.31.14:4664: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49346->77.90.31.14:4664: use of closed network connection
Jan 1 08:04:58.839161787 (640h35m58.710244842s) [noise=true , address=77.90.31.14:4664 , piece_id=CAGDEGYB2537JB3VPBX5TSWT3Q6NF66TVBETJGGWJERJRH3L5A2A, seg_pos={0 0}, upload=561312]: Long tail cancellation due to optimal threshold
Jan 1 08:04:58.839181845 (640h35m58.71026488s) [seg_pos={0 0}, upload=561312]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 1 08:04:58.839260786 (640h35m58.710343831s) [upload=561312]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 1 08:04:58.839265575 (640h35m58.71034861s) [upload=561312]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 1 08:04:59.146721350 (640h35m59.017804385s) [upload=561312]: Done uploading segments...
Jan 1 08:04:59.146740697 (640h35m59.017823732s) [upload=561312]: Upload finished. err: <nil>
Duration: 30.003337017s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [.............X.....X..XX.........................XX.........] duration:
33.272945088s trace:
https://jaeger.dev.storj.io/trace/10d96081b66cd020
FAIL (2026-01-01 08:05:24 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
73329s) [noise=true , address=145.79.227.40:4066 , piece_id=ABZ7356MEBO4XPYAKFSS4EQJYJN4CWNJR3C6NXFWLAEO65KL3J5Q, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.722330803 (640h29m1.593413848s) [noise=true , address=145.79.227.9:4391 , piece_id=XS6HYQTJO355YLXEIDBKHTJ44UAT5UBKHDUCFA744X6TBPGYN5YQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.722848709 (640h29m1.593931754s) [noise=true , address=77.90.31.12:4606 , piece_id=K4E5ECNMOMKEIFHZ3EZ2QSQHZ5IYP3CAD5DA6DBQLM4JYMOYU7MA, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.723452529 (640h29m1.594535574s) [noise=true , address=193.149.164.143:4694 , piece_id=EUO4UWMROIFFAJECCFRJBQF4XV25VDKGKX3UF2PTUMTE5JBGD67Q, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.723730728 (640h29m1.594813773s) [noise=true , address=45.91.116.5:4320 , piece_id=FOFLP7277VUUPTRZZHQLNVJ3YVYBELJTUYL7PTT7FJNN7J7QLS2A, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.724307808 (640h29m1.595390853s) [noise=true , address=64.49.13.169:4705 , piece_id=2P3JANSNK64726X7IYJOIQOKAQ4P4LJ5A2GVPX6UXOWUDIYRPZMA, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.724847144 (640h29m1.595930189s) [noise=true , address=64.49.13.137:4494 , piece_id=GCFOH6XRKDEGNG6TFPCMHG5RUCP2LUFS3XEOPNF42VYTGS63ZHQQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.724970960 (640h29m1.596054005s) [noise=true , address=77.90.31.15:4725 , piece_id=LLGZBPL7CHDI2TAY2SYMUMC264USSUQRVFR5P4HTLBWPBOTSUSOA, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.725289036 (640h29m1.596372071s) [noise=true , address=145.79.227.49:4371 , piece_id=RGZZMDFFVVSIQJ2ZEUSK275SBWMGV75WGNDLY6LJ2TNRPUSY7RMQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.725518012 (640h29m1.596601047s) [noise=true , address=77.90.31.13:4649 , piece_id=YRKZDAT5O2USQM4SO5GRZ45SX7MA3U52KN2CRGYPX5WJ4HIKOJBA, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.725902264 (640h29m1.596985309s) [noise=true , address=45.91.116.12:4238 , piece_id=R5LPEYTYC3XB76SSMDVHPZORAWB2P3S4TETMYV23NJI4UXYUYCEQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.726176987 (640h29m1.597260032s) [noise=true , address=145.79.227.35:4497 , piece_id=YDNRFAJVXHRJD5QHICHR6CGF6K5ARZTLHOUSRLV6JAYU3ZSWEY3A, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.726843697 (640h29m1.597926742s) [noise=true , address=77.90.31.19:4846 , piece_id=2XJSN2WTR2BAVKDJZXEMY6SRX7KLOU4SVGYH7TNJIJTPW723DEHQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.727470170 (640h29m1.598553225s) [noise=true , address=145.79.227.30:4784 , piece_id=HA6Z6M3HASDLIDP2PHS7MUDOHBF6T6ZNPTABXF5PYXR2HFYSBOPQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.728088075 (640h29m1.59917112s) [noise=true , address=145.79.227.39:4032 , piece_id=Z7BQPDXOVQ6AEKACBB2DJFI652AT6UIHXUB2HPUJFMK53EF73AWQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:03.152355605 (640h29m3.02343864s) [upload=561203]: Next returned nil segment
Jan 1 07:58:03.152371765 (640h29m3.023454811s) [upload=561203]: Waiting for error group managing segments...
Jan 1 07:58:03.563991563 (640h29m3.435074608s) [noise=true , address=145.79.227.9:4391 , piece_id=XS6HYQTJO355YLXEIDBKHTJ44UAT5UBKHDUCFA744X6TBPGYN5YQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:03.652285026 (640h29m3.523368071s) [noise=true , address=145.79.227.49:4371 , piece_id=RGZZMDFFVVSIQJ2ZEUSK275SBWMGV75WGNDLY6LJ2TNRPUSY7RMQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:03.671359306 (640h29m3.542442351s) [noise=true , address=145.79.227.39:4032 , piece_id=Z7BQPDXOVQ6AEKACBB2DJFI652AT6UIHXUB2HPUJFMK53EF73AWQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:03.774991565 (640h29m3.64607462s) [noise=true , address=77.90.31.19:4846 , piece_id=2XJSN2WTR2BAVKDJZXEMY6SRX7KLOU4SVGYH7TNJIJTPW723DEHQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:03.870781221 (640h29m3.741864266s) [noise=true , address=77.90.31.12:4606 , piece_id=K4E5ECNMOMKEIFHZ3EZ2QSQHZ5IYP3CAD5DA6DBQLM4JYMOYU7MA, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:03.878179488 (640h29m3.749262533s) [noise=true , address=77.90.31.13:4649 , piece_id=YRKZDAT5O2USQM4SO5GRZ45SX7MA3U52KN2CRGYPX5WJ4HIKOJBA, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:03.977331467 (640h29m3.848414512s) [noise=true , address=77.90.31.15:4725 , piece_id=LLGZBPL7CHDI2TAY2SYMUMC264USSUQRVFR5P4HTLBWPBOTSUSOA, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.224768738 (640h29m4.095851783s) [noise=true , address=45.91.116.12:4238 , piece_id=R5LPEYTYC3XB76SSMDVHPZORAWB2P3S4TETMYV23NJI4UXYUYCEQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.457462948 (640h29m4.328545994s) [noise=true , address=45.91.116.5:4320 , piece_id=FOFLP7277VUUPTRZZHQLNVJ3YVYBELJTUYL7PTT7FJNN7J7QLS2A, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.488291199 (640h29m4.359374234s) [noise=true , address=145.79.227.35:4497 , piece_id=YDNRFAJVXHRJD5QHICHR6CGF6K5ARZTLHOUSRLV6JAYU3ZSWEY3A, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.522541837 (640h29m4.393624882s) [noise=true , address=145.79.227.43:4171 , piece_id=KYGE2RFI6O6DUFWVGJV2IT5VE3U43WJ4ODNQMTZFQH2DAADFC6BQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.525844266 (640h29m4.396927321s) [noise=true , address=145.79.227.33:4886 , piece_id=CYGCX4TM2ZFPCJYVDF52PDNR2OCNZ2NUSUUOUMUHYGCADWWFDHVQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.539045983 (640h29m4.410129028s) [noise=true , address=145.79.227.34:4924 , piece_id=QUPSGORFPMQVD2IRAK7COO2EE7XDV2KWEBTI2FQ2C25RTAAFNSRA, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.544558039 (640h29m4.415641084s) [noise=true , address=145.79.227.30:4784 , piece_id=HA6Z6M3HASDLIDP2PHS7MUDOHBF6T6ZNPTABXF5PYXR2HFYSBOPQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.544582956 (640h29m4.415665991s) [seg_pos={0 0}, upload=561203]: Segment reached optimal threshold of 14 pieces.
Jan 1 07:58:04.544811462 (640h29m4.415894527s) [noise=true , address=193.149.164.143:4694 , piece_id=EUO4UWMROIFFAJECCFRJBQF4XV25VDKGKX3UF2PTUMTE5JBGD67Q, seg_pos={0 0}, upload=561203]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx): protocol: expected piece hash; context canceled; EOF
Jan 1 07:58:04.544821240 (640h29m4.415904275s) [noise=true , address=193.149.164.143:4694 , piece_id=EUO4UWMROIFFAJECCFRJBQF4XV25VDKGKX3UF2PTUMTE5JBGD67Q, seg_pos={0 0}, upload=561203]: Long tail cancellation due to optimal threshold
Jan 1 07:58:04.545116732 (640h29m4.416199767s) [noise=true , address=64.49.13.169:4705 , piece_id=2P3JANSNK64726X7IYJOIQOKAQ4P4LJ5A2GVPX6UXOWUDIYRPZMA, seg_pos={0 0}, upload=561203]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:45312->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:45312->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45312->64.49.13.169:4705: use of closed network connection
Jan 1 07:58:04.545123666 (640h29m4.416206711s) [noise=true , address=64.49.13.169:4705 , piece_id=2P3JANSNK64726X7IYJOIQOKAQ4P4LJ5A2GVPX6UXOWUDIYRPZMA, seg_pos={0 0}, upload=561203]: Long tail cancellation due to optimal threshold
Jan 1 07:58:04.545303088 (640h29m4.416386123s) [noise=true , address=145.79.227.40:4066 , piece_id=ABZ7356MEBO4XPYAKFSS4EQJYJN4CWNJR3C6NXFWLAEO65KL3J5Q, seg_pos={0 0}, upload=561203]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF
Jan 1 07:58:04.545312055 (640h29m4.4163951s) [noise=true , address=145.79.227.40:4066 , piece_id=ABZ7356MEBO4XPYAKFSS4EQJYJN4CWNJR3C6NXFWLAEO65KL3J5Q, seg_pos={0 0}, upload=561203]: Long tail cancellation due to optimal threshold
Jan 1 07:58:04.545414279 (640h29m4.416497314s) [noise=true , address=64.49.13.137:4494 , piece_id=GCFOH6XRKDEGNG6TFPCMHG5RUCP2LUFS3XEOPNF42VYTGS63ZHQQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TX66fDVv3qu4yBGx9FNZTiArNeUnqxjMYwb3VGqYCQRe8RwoU): protocol: write tcp 188.166.225.170:35392->64.49.13.137:4494: use of closed network connection; write tcp 188.166.225.170:35392->64.49.13.137:4494: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35392->64.49.13.137:4494: use of closed network connection
Jan 1 07:58:04.545422815 (640h29m4.41650585s) [noise=true , address=64.49.13.137:4494 , piece_id=GCFOH6XRKDEGNG6TFPCMHG5RUCP2LUFS3XEOPNF42VYTGS63ZHQQ, seg_pos={0 0}, upload=561203]: Long tail cancellation due to optimal threshold
Jan 1 07:58:04.545439939 (640h29m4.416522974s) [seg_pos={0 0}, upload=561203]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 1 07:58:04.545580406 (640h29m4.416663441s) [upload=561203]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 1 07:58:04.545587800 (640h29m4.416670845s) [upload=561203]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 1 07:58:04.840726371 (640h29m4.711809416s) [upload=561203]: Done uploading segments...
Jan 1 07:58:04.840749646 (640h29m4.711832691s) [upload=561203]: Upload finished. err: <nil>
Duration: 33.272969976s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
16.260922741s trace:
https://jaeger.dev.storj.io/trace/44e9d74e3b154e8e
FAIL (2026-01-01 08:05:07 UTC)
orbiter warning: context deadline exceeded
Duration: 16.260951485s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
38 out of 60 is failed (threshold=6) [.XX.X.XXXXX.XX..XXXXX.XXXX..XX.XXX.XX...X.X..X...XXXXXX..XXX] duration:
5.001777599s trace:
https://jaeger.dev.storj.io/trace/336092c2ca3e559b
FAIL (2026-01-01 08:04:44 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001799922s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [.X.XX.XX..X...X.XX.X...X.........X.X........XX.X...X..X....X] duration:
30.002260507s trace:
https://jaeger.dev.storj.io/trace/5071b6f2fa7b8236
FAIL (2026-01-01 08:04:20 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: QZ6RKPNNEDLRKOPRBCLCF7RZKNGO4AR3UIW4AZ5ZFL5JDH2QMLIQ) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: VXRJJOOITNMPYEX37EUVP4VDJ2Z7IU7M76QQ4XNDZRXU3RK7JITQ) context deadline exceeded
======LOGS======
Jan 1 08:03:51.018112620 (640h34m50.889195665s) [upload=561295]: Starting upload
Jan 1 08:03:51.018173425 (640h34m50.88925646s) [upload=561295]: Done starting upload
Jan 1 08:03:51.018237527 (640h34m50.889320562s) [upload=561295]: Uploading segments...
Jan 1 08:03:51.018400879 (640h34m50.889483924s) [upload=561295]: Got next segment. Inline: false
Jan 1 08:03:51.018437939 (640h34m50.889520984s) [upload=561295]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 1 08:03:51.018443490 (640h34m50.889526535s) [upload=561295]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 1 08:03:51.267840284 (640h34m51.138923369s) [seg_pos={0 0}, upload=561295]: Begin upload segment...
Jan 1 08:03:51.267903143 (640h34m51.138986188s) [seg_pos={0 0}, upload=561295]: Done begin upload segment.
Jan 1 08:03:51.267981052 (640h34m51.139064097s) [noise=true , address=145.79.227.27:4690 , piece_id=2SBTZAHHIYSNENYCRBUDZFPMWNQ2VJSD6NTRVUB5MPZSRXYIKXYQ, seg_pos={0 0}, upload=561295]: Uploading piece...
Jan 1 08:03:51.268010978 (640h34m51.139094023s) [noise=true , address=145.79.227.17:4971 , piece_id=6SE4UCPN2HAF5J45IRCZHBG44YTBICGEWOV4RV57OADAJUXQU7VQ, seg_pos={0 0}, upload=561295]: Uploading piece...
Jan 1 08:03:51.268159171 (640h34m51.139242206s) [noise=true , address=45.91.116.11:4117 , piece_id=QWDKF6P7XD6IPH44YMGX5AHNO7D7GRSNFNAA7EIY732ERFGNHDJA, seg_pos={0 0}, upload=561295]: Uploading piece...
Jan 1 08:03:51.268598847 (640h34m51.139681893s) [noise=true , address=145.79.227.16:4909 , piece_id=UGA5NEJEN4QYFQOII2T6FI6N233KRYZYRY7AETHXSC5ZCS2KVXGQ, seg_pos={0 0}, upload=561295]: Uploading piece...
Jan 1 08:03:51.269018046 (640h34m51.140101091s) [noise=true , address=145.79.227.14:4838 , piece_id=HSBCLQLQXP5FWDDCLS4YFCUUJRLJH74TX5TGNGU35FPDLT5JX5PA, seg_pos={0 0}, upload=561295]: Uploading piece...
Jan 1 08:03:51.269460769 (640h34m51.140543814s) [noise=true , address=145.79.227.43:4159 , piece_id=CPUW3MXRCUA74PAHOBEDTQNQXDDAJ46P4QDT5MRZTQKSM32WHHFA, seg_pos={0 0}, upload=561295]: Uploading piece...
Jan 1 08:03:51.270046324 (640h34m51.141129369s) [noise=true , address=145.79.227.45:4231 , piece_id=CGOX2GK2PPIFSAPLKR5G5PBKUBY5GJYVUWUVQYYMXXSZZRJSUBSQ, seg_pos={0 0}, upload=561295]: Uploading piece...
Jan 1 08:03:51.270675071 (640h34m51.141758126s) [noise=true , address=145.79.227.4:4187 , piece_id=QQVMY72QBWQMTBVSKGVRTXV4D2U5GBE3ASIEZJEBCDTJBFXDQOHA, seg_pos={0 0}, upload=561295]: Uploading piece...
Jan 1 08:03:51.271040677 (640h34m51.142123722s) [noise=true , address=145.79.227.41:4086 , piece_id=S5PX6K3EHWKLWSCF3OFVFHQ6GI5BZ457XZ6FOXVIHQAZ4T3OY6EA, seg_pos={0 0}, upload=561295]: Uploading piece...
Jan 1 08:03:51.271071305 (640h34m51.14215434s) [noise=true , address=145.79.227.21:4473 , piece_id=TEJHNGG7TBYMST7ZPUMVNGRHKSKP6AGZEWBPIOA2MKOC3QV62QTQ, seg_pos={0 0}, upload=561295]: Uploading piece...
Jan 1 08:03:51.271520882 (640h34m51.142603917s) [noise=true , address=45.91.116.14:4189 , piece_id=ODQFNEQS6ZRVZMF3HULD4IOTR7PIRKDLOKZ7BHXVUBBEQKCP5RAQ, seg_pos={0 0}, upload=561295]: Uploading piece...
Jan 1 08:03:51.271836593 (640h34m51.142919638s) [noise=true , address=77.90.31.10:4519 , piece_id=VXRJJOOITNMPYEX37EUVP4VDJ2Z7IU7M76QQ4XNDZRXU3RK7JITQ, seg_pos={0 0}, upload=561295]: Uploading piece...
Jan 1 08:03:51.272019721 (640h34m51.143102756s) [noise=true , address=145.79.227.9:4366 , piece_id=TAB4VNARBCZBPDCSI6R3EZE5TUIWGRXLOA6TF5YHMA2WWI2HMZ5Q, seg_pos={0 0}, upload=561295]: Uploading piece...
Jan 1 08:03:51.272468456 (640h34m51.143551501s) [noise=true , address=77.90.31.19:4841 , piece_id=QZ6RKPNNEDLRKOPRBCLCF7RZKNGO4AR3UIW4AZ5ZFL5JDH2QMLIQ, seg_pos={0 0}, upload=561295]: Uploading piece...
Jan 1 08:03:51.272658999 (640h34m51.143742044s) [noise=true , address=145.79.227.6:4257 , piece_id=54UD5PCAOH2GMT765X34G2YDTKKYPNUEUVRZUIBUF7JDU42OFKIQ, seg_pos={0 0}, upload=561295]: Uploading piece...
Jan 1 08:03:51.273160824 (640h34m51.144243869s) [noise=true , address=145.79.227.48:4301 , piece_id=VQJROCHCQ2YBQ52RAUCAG2YJCT6EDP4IPXPIMHYPFRABLQCQDO4Q, seg_pos={0 0}, upload=561295]: Uploading piece...
Jan 1 08:03:52.972964531 (640h34m52.844047576s) [upload=561295]: Next returned nil segment
Jan 1 08:03:52.972979210 (640h34m52.844062255s) [upload=561295]: Waiting for error group managing segments...
Jan 1 08:03:53.259617525 (640h34m53.13070056s) [noise=true , address=145.79.227.48:4301 , piece_id=VQJROCHCQ2YBQ52RAUCAG2YJCT6EDP4IPXPIMHYPFRABLQCQDO4Q, seg_pos={0 0}, upload=561295]: Done uploading piece. err: <nil>
Jan 1 08:03:53.262345971 (640h34m53.133429006s) [noise=true , address=145.79.227.21:4473 , piece_id=TEJHNGG7TBYMST7ZPUMVNGRHKSKP6AGZEWBPIOA2MKOC3QV62QTQ, seg_pos={0 0}, upload=561295]: Done uploading piece. err: <nil>
Jan 1 08:03:53.269300714 (640h34m53.140383749s) [noise=true , address=145.79.227.14:4838 , piece_id=HSBCLQLQXP5FWDDCLS4YFCUUJRLJH74TX5TGNGU35FPDLT5JX5PA, seg_pos={0 0}, upload=561295]: Done uploading piece. err: <nil>
Jan 1 08:03:53.275775052 (640h34m53.146858097s) [noise=true , address=145.79.227.16:4909 , piece_id=UGA5NEJEN4QYFQOII2T6FI6N233KRYZYRY7AETHXSC5ZCS2KVXGQ, seg_pos={0 0}, upload=561295]: Done uploading piece. err: <nil>
Jan 1 08:03:53.276489241 (640h34m53.147572286s) [noise=true , address=145.79.227.17:4971 , piece_id=6SE4UCPN2HAF5J45IRCZHBG44YTBICGEWOV4RV57OADAJUXQU7VQ, seg_pos={0 0}, upload=561295]: Done uploading piece. err: <nil>
Jan 1 08:03:53.287993156 (640h34m53.159076201s) [noise=true , address=145.79.227.45:4231 , piece_id=CGOX2GK2PPIFSAPLKR5G5PBKUBY5GJYVUWUVQYYMXXSZZRJSUBSQ, seg_pos={0 0}, upload=561295]: Done uploading piece. err: <nil>
Jan 1 08:03:53.290630158 (640h34m53.161713213s) [noise=true , address=145.79.227.27:4690 , piece_id=2SBTZAHHIYSNENYCRBUDZFPMWNQ2VJSD6NTRVUB5MPZSRXYIKXYQ, seg_pos={0 0}, upload=561295]: Done uploading piece. err: <nil>
Jan 1 08:03:53.291189493 (640h34m53.162272538s) [noise=true , address=145.79.227.9:4366 , piece_id=TAB4VNARBCZBPDCSI6R3EZE5TUIWGRXLOA6TF5YHMA2WWI2HMZ5Q, seg_pos={0 0}, upload=561295]: Done uploading piece. err: <nil>
Jan 1 08:03:53.349861732 (640h34m53.220944788s) [noise=true , address=145.79.227.41:4086 , piece_id=S5PX6K3EHWKLWSCF3OFVFHQ6GI5BZ457XZ6FOXVIHQAZ4T3OY6EA, seg_pos={0 0}, upload=561295]: Done uploading piece. err: <nil>
Jan 1 08:03:53.361667883 (640h34m53.232750918s) [noise=true , address=145.79.227.4:4187 , piece_id=QQVMY72QBWQMTBVSKGVRTXV4D2U5GBE3ASIEZJEBCDTJBFXDQOHA, seg_pos={0 0}, upload=561295]: Done uploading piece. err: <nil>
Jan 1 08:03:53.370688950 (640h34m53.241771995s) [noise=true , address=145.79.227.6:4257 , piece_id=54UD5PCAOH2GMT765X34G2YDTKKYPNUEUVRZUIBUF7JDU42OFKIQ, seg_pos={0 0}, upload=561295]: Done uploading piece. err: <nil>
Jan 1 08:03:53.544681845 (640h34m53.41576489s) [noise=true , address=77.90.31.10:4519 , piece_id=VXRJJOOITNMPYEX37EUVP4VDJ2Z7IU7M76QQ4XNDZRXU3RK7JITQ, seg_pos={0 0}, upload=561295]: Done uploading piece. err: <nil>
Jan 1 08:03:53.554493047 (640h34m53.425576083s) [noise=true , address=77.90.31.19:4841 , piece_id=QZ6RKPNNEDLRKOPRBCLCF7RZKNGO4AR3UIW4AZ5ZFL5JDH2QMLIQ, seg_pos={0 0}, upload=561295]: Done uploading piece. err: <nil>
Jan 1 08:03:53.554524748 (640h34m53.425607783s) [seg_pos={0 0}, upload=561295]: Segment reached optimal threshold of 13 pieces.
Jan 1 08:03:53.554772870 (640h34m53.425855915s) [noise=true , address=145.79.227.43:4159 , piece_id=CPUW3MXRCUA74PAHOBEDTQNQXDDAJ46P4QDT5MRZTQKSM32WHHFA, seg_pos={0 0}, upload=561295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF
Jan 1 08:03:53.554782438 (640h34m53.425865473s) [noise=true , address=145.79.227.43:4159 , piece_id=CPUW3MXRCUA74PAHOBEDTQNQXDDAJ46P4QDT5MRZTQKSM32WHHFA, seg_pos={0 0}, upload=561295]: Long tail cancellation due to optimal threshold
Jan 1 08:03:53.554913628 (640h34m53.425996703s) [noise=true , address=45.91.116.11:4117 , piece_id=QWDKF6P7XD6IPH44YMGX5AHNO7D7GRSNFNAA7EIY732ERFGNHDJA, seg_pos={0 0}, upload=561295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Jan 1 08:03:53.554942854 (640h34m53.426025899s) [noise=true , address=45.91.116.11:4117 , piece_id=QWDKF6P7XD6IPH44YMGX5AHNO7D7GRSNFNAA7EIY732ERFGNHDJA, seg_pos={0 0}, upload=561295]: Long tail cancellation due to optimal threshold
Jan 1 08:03:53.555077379 (640h34m53.426160414s) [noise=true , address=45.91.116.14:4189 , piece_id=ODQFNEQS6ZRVZMF3HULD4IOTR7PIRKDLOKZ7BHXVUBBEQKCP5RAQ, seg_pos={0 0}, upload=561295]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF
Jan 1 08:03:53.555085816 (640h34m53.426168851s) [noise=true , address=45.91.116.14:4189 , piece_id=ODQFNEQS6ZRVZMF3HULD4IOTR7PIRKDLOKZ7BHXVUBBEQKCP5RAQ, seg_pos={0 0}, upload=561295]: Long tail cancellation due to optimal threshold
Jan 1 08:03:53.555111064 (640h34m53.426194099s) [seg_pos={0 0}, upload=561295]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 1 08:03:53.555198340 (640h34m53.426281385s) [upload=561295]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 1 08:03:53.555203800 (640h34m53.426286846s) [upload=561295]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 1 08:03:53.841983375 (640h34m53.71306642s) [upload=561295]: Done uploading segments...
Jan 1 08:03:53.842009054 (640h34m53.713092089s) [upload=561295]: Upload finished. err: <nil>
Duration: 30.00227227s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [..............X.....X..XX.........................XX........] duration:
31.600938228s trace:
https://jaeger.dev.storj.io/trace/10d96081b66cd020
FAIL (2026-01-01 08:04:12 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
73329s) [noise=true , address=145.79.227.40:4066 , piece_id=ABZ7356MEBO4XPYAKFSS4EQJYJN4CWNJR3C6NXFWLAEO65KL3J5Q, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.722330803 (640h29m1.593413848s) [noise=true , address=145.79.227.9:4391 , piece_id=XS6HYQTJO355YLXEIDBKHTJ44UAT5UBKHDUCFA744X6TBPGYN5YQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.722848709 (640h29m1.593931754s) [noise=true , address=77.90.31.12:4606 , piece_id=K4E5ECNMOMKEIFHZ3EZ2QSQHZ5IYP3CAD5DA6DBQLM4JYMOYU7MA, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.723452529 (640h29m1.594535574s) [noise=true , address=193.149.164.143:4694 , piece_id=EUO4UWMROIFFAJECCFRJBQF4XV25VDKGKX3UF2PTUMTE5JBGD67Q, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.723730728 (640h29m1.594813773s) [noise=true , address=45.91.116.5:4320 , piece_id=FOFLP7277VUUPTRZZHQLNVJ3YVYBELJTUYL7PTT7FJNN7J7QLS2A, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.724307808 (640h29m1.595390853s) [noise=true , address=64.49.13.169:4705 , piece_id=2P3JANSNK64726X7IYJOIQOKAQ4P4LJ5A2GVPX6UXOWUDIYRPZMA, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.724847144 (640h29m1.595930189s) [noise=true , address=64.49.13.137:4494 , piece_id=GCFOH6XRKDEGNG6TFPCMHG5RUCP2LUFS3XEOPNF42VYTGS63ZHQQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.724970960 (640h29m1.596054005s) [noise=true , address=77.90.31.15:4725 , piece_id=LLGZBPL7CHDI2TAY2SYMUMC264USSUQRVFR5P4HTLBWPBOTSUSOA, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.725289036 (640h29m1.596372071s) [noise=true , address=145.79.227.49:4371 , piece_id=RGZZMDFFVVSIQJ2ZEUSK275SBWMGV75WGNDLY6LJ2TNRPUSY7RMQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.725518012 (640h29m1.596601047s) [noise=true , address=77.90.31.13:4649 , piece_id=YRKZDAT5O2USQM4SO5GRZ45SX7MA3U52KN2CRGYPX5WJ4HIKOJBA, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.725902264 (640h29m1.596985309s) [noise=true , address=45.91.116.12:4238 , piece_id=R5LPEYTYC3XB76SSMDVHPZORAWB2P3S4TETMYV23NJI4UXYUYCEQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.726176987 (640h29m1.597260032s) [noise=true , address=145.79.227.35:4497 , piece_id=YDNRFAJVXHRJD5QHICHR6CGF6K5ARZTLHOUSRLV6JAYU3ZSWEY3A, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.726843697 (640h29m1.597926742s) [noise=true , address=77.90.31.19:4846 , piece_id=2XJSN2WTR2BAVKDJZXEMY6SRX7KLOU4SVGYH7TNJIJTPW723DEHQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.727470170 (640h29m1.598553225s) [noise=true , address=145.79.227.30:4784 , piece_id=HA6Z6M3HASDLIDP2PHS7MUDOHBF6T6ZNPTABXF5PYXR2HFYSBOPQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.728088075 (640h29m1.59917112s) [noise=true , address=145.79.227.39:4032 , piece_id=Z7BQPDXOVQ6AEKACBB2DJFI652AT6UIHXUB2HPUJFMK53EF73AWQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:03.152355605 (640h29m3.02343864s) [upload=561203]: Next returned nil segment
Jan 1 07:58:03.152371765 (640h29m3.023454811s) [upload=561203]: Waiting for error group managing segments...
Jan 1 07:58:03.563991563 (640h29m3.435074608s) [noise=true , address=145.79.227.9:4391 , piece_id=XS6HYQTJO355YLXEIDBKHTJ44UAT5UBKHDUCFA744X6TBPGYN5YQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:03.652285026 (640h29m3.523368071s) [noise=true , address=145.79.227.49:4371 , piece_id=RGZZMDFFVVSIQJ2ZEUSK275SBWMGV75WGNDLY6LJ2TNRPUSY7RMQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:03.671359306 (640h29m3.542442351s) [noise=true , address=145.79.227.39:4032 , piece_id=Z7BQPDXOVQ6AEKACBB2DJFI652AT6UIHXUB2HPUJFMK53EF73AWQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:03.774991565 (640h29m3.64607462s) [noise=true , address=77.90.31.19:4846 , piece_id=2XJSN2WTR2BAVKDJZXEMY6SRX7KLOU4SVGYH7TNJIJTPW723DEHQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:03.870781221 (640h29m3.741864266s) [noise=true , address=77.90.31.12:4606 , piece_id=K4E5ECNMOMKEIFHZ3EZ2QSQHZ5IYP3CAD5DA6DBQLM4JYMOYU7MA, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:03.878179488 (640h29m3.749262533s) [noise=true , address=77.90.31.13:4649 , piece_id=YRKZDAT5O2USQM4SO5GRZ45SX7MA3U52KN2CRGYPX5WJ4HIKOJBA, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:03.977331467 (640h29m3.848414512s) [noise=true , address=77.90.31.15:4725 , piece_id=LLGZBPL7CHDI2TAY2SYMUMC264USSUQRVFR5P4HTLBWPBOTSUSOA, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.224768738 (640h29m4.095851783s) [noise=true , address=45.91.116.12:4238 , piece_id=R5LPEYTYC3XB76SSMDVHPZORAWB2P3S4TETMYV23NJI4UXYUYCEQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.457462948 (640h29m4.328545994s) [noise=true , address=45.91.116.5:4320 , piece_id=FOFLP7277VUUPTRZZHQLNVJ3YVYBELJTUYL7PTT7FJNN7J7QLS2A, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.488291199 (640h29m4.359374234s) [noise=true , address=145.79.227.35:4497 , piece_id=YDNRFAJVXHRJD5QHICHR6CGF6K5ARZTLHOUSRLV6JAYU3ZSWEY3A, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.522541837 (640h29m4.393624882s) [noise=true , address=145.79.227.43:4171 , piece_id=KYGE2RFI6O6DUFWVGJV2IT5VE3U43WJ4ODNQMTZFQH2DAADFC6BQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.525844266 (640h29m4.396927321s) [noise=true , address=145.79.227.33:4886 , piece_id=CYGCX4TM2ZFPCJYVDF52PDNR2OCNZ2NUSUUOUMUHYGCADWWFDHVQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.539045983 (640h29m4.410129028s) [noise=true , address=145.79.227.34:4924 , piece_id=QUPSGORFPMQVD2IRAK7COO2EE7XDV2KWEBTI2FQ2C25RTAAFNSRA, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.544558039 (640h29m4.415641084s) [noise=true , address=145.79.227.30:4784 , piece_id=HA6Z6M3HASDLIDP2PHS7MUDOHBF6T6ZNPTABXF5PYXR2HFYSBOPQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.544582956 (640h29m4.415665991s) [seg_pos={0 0}, upload=561203]: Segment reached optimal threshold of 14 pieces.
Jan 1 07:58:04.544811462 (640h29m4.415894527s) [noise=true , address=193.149.164.143:4694 , piece_id=EUO4UWMROIFFAJECCFRJBQF4XV25VDKGKX3UF2PTUMTE5JBGD67Q, seg_pos={0 0}, upload=561203]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx): protocol: expected piece hash; context canceled; EOF
Jan 1 07:58:04.544821240 (640h29m4.415904275s) [noise=true , address=193.149.164.143:4694 , piece_id=EUO4UWMROIFFAJECCFRJBQF4XV25VDKGKX3UF2PTUMTE5JBGD67Q, seg_pos={0 0}, upload=561203]: Long tail cancellation due to optimal threshold
Jan 1 07:58:04.545116732 (640h29m4.416199767s) [noise=true , address=64.49.13.169:4705 , piece_id=2P3JANSNK64726X7IYJOIQOKAQ4P4LJ5A2GVPX6UXOWUDIYRPZMA, seg_pos={0 0}, upload=561203]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:45312->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:45312->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45312->64.49.13.169:4705: use of closed network connection
Jan 1 07:58:04.545123666 (640h29m4.416206711s) [noise=true , address=64.49.13.169:4705 , piece_id=2P3JANSNK64726X7IYJOIQOKAQ4P4LJ5A2GVPX6UXOWUDIYRPZMA, seg_pos={0 0}, upload=561203]: Long tail cancellation due to optimal threshold
Jan 1 07:58:04.545303088 (640h29m4.416386123s) [noise=true , address=145.79.227.40:4066 , piece_id=ABZ7356MEBO4XPYAKFSS4EQJYJN4CWNJR3C6NXFWLAEO65KL3J5Q, seg_pos={0 0}, upload=561203]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF
Jan 1 07:58:04.545312055 (640h29m4.4163951s) [noise=true , address=145.79.227.40:4066 , piece_id=ABZ7356MEBO4XPYAKFSS4EQJYJN4CWNJR3C6NXFWLAEO65KL3J5Q, seg_pos={0 0}, upload=561203]: Long tail cancellation due to optimal threshold
Jan 1 07:58:04.545414279 (640h29m4.416497314s) [noise=true , address=64.49.13.137:4494 , piece_id=GCFOH6XRKDEGNG6TFPCMHG5RUCP2LUFS3XEOPNF42VYTGS63ZHQQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TX66fDVv3qu4yBGx9FNZTiArNeUnqxjMYwb3VGqYCQRe8RwoU): protocol: write tcp 188.166.225.170:35392->64.49.13.137:4494: use of closed network connection; write tcp 188.166.225.170:35392->64.49.13.137:4494: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35392->64.49.13.137:4494: use of closed network connection
Jan 1 07:58:04.545422815 (640h29m4.41650585s) [noise=true , address=64.49.13.137:4494 , piece_id=GCFOH6XRKDEGNG6TFPCMHG5RUCP2LUFS3XEOPNF42VYTGS63ZHQQ, seg_pos={0 0}, upload=561203]: Long tail cancellation due to optimal threshold
Jan 1 07:58:04.545439939 (640h29m4.416522974s) [seg_pos={0 0}, upload=561203]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 1 07:58:04.545580406 (640h29m4.416663441s) [upload=561203]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 1 07:58:04.545587800 (640h29m4.416670845s) [upload=561203]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 1 07:58:04.840726371 (640h29m4.711809416s) [upload=561203]: Done uploading segments...
Jan 1 07:58:04.840749646 (640h29m4.711832691s) [upload=561203]: Upload finished. err: <nil>
Duration: 31.600952685s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................] duration:
12.413781397s trace:
https://jaeger.dev.storj.io/trace/44e9d74e3b154e8e
FAIL (2026-01-01 08:03:52 UTC)
orbiter warning: context deadline exceeded
Duration: 12.413801706s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
38 out of 60 is failed (threshold=6) [X.XX.X.XXXXX.XX..XXXXX.XXXX..XX.XXX.XX...X.X..X...XXXXXX..XX] duration:
5.00047941s trace:
https://jaeger.dev.storj.io/trace/2847b93772ae373b
FAIL (2026-01-01 08:03:49 UTC)
orbiter warning: context deadline exceeded
Duration: 5.00049541s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [...............X.....X..XX.........................XX.......] duration:
36.660563023s trace:
https://jaeger.dev.storj.io/trace/10d96081b66cd020
FAIL (2026-01-01 08:03:40 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
73329s) [noise=true , address=145.79.227.40:4066 , piece_id=ABZ7356MEBO4XPYAKFSS4EQJYJN4CWNJR3C6NXFWLAEO65KL3J5Q, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.722330803 (640h29m1.593413848s) [noise=true , address=145.79.227.9:4391 , piece_id=XS6HYQTJO355YLXEIDBKHTJ44UAT5UBKHDUCFA744X6TBPGYN5YQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.722848709 (640h29m1.593931754s) [noise=true , address=77.90.31.12:4606 , piece_id=K4E5ECNMOMKEIFHZ3EZ2QSQHZ5IYP3CAD5DA6DBQLM4JYMOYU7MA, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.723452529 (640h29m1.594535574s) [noise=true , address=193.149.164.143:4694 , piece_id=EUO4UWMROIFFAJECCFRJBQF4XV25VDKGKX3UF2PTUMTE5JBGD67Q, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.723730728 (640h29m1.594813773s) [noise=true , address=45.91.116.5:4320 , piece_id=FOFLP7277VUUPTRZZHQLNVJ3YVYBELJTUYL7PTT7FJNN7J7QLS2A, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.724307808 (640h29m1.595390853s) [noise=true , address=64.49.13.169:4705 , piece_id=2P3JANSNK64726X7IYJOIQOKAQ4P4LJ5A2GVPX6UXOWUDIYRPZMA, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.724847144 (640h29m1.595930189s) [noise=true , address=64.49.13.137:4494 , piece_id=GCFOH6XRKDEGNG6TFPCMHG5RUCP2LUFS3XEOPNF42VYTGS63ZHQQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.724970960 (640h29m1.596054005s) [noise=true , address=77.90.31.15:4725 , piece_id=LLGZBPL7CHDI2TAY2SYMUMC264USSUQRVFR5P4HTLBWPBOTSUSOA, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.725289036 (640h29m1.596372071s) [noise=true , address=145.79.227.49:4371 , piece_id=RGZZMDFFVVSIQJ2ZEUSK275SBWMGV75WGNDLY6LJ2TNRPUSY7RMQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.725518012 (640h29m1.596601047s) [noise=true , address=77.90.31.13:4649 , piece_id=YRKZDAT5O2USQM4SO5GRZ45SX7MA3U52KN2CRGYPX5WJ4HIKOJBA, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.725902264 (640h29m1.596985309s) [noise=true , address=45.91.116.12:4238 , piece_id=R5LPEYTYC3XB76SSMDVHPZORAWB2P3S4TETMYV23NJI4UXYUYCEQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.726176987 (640h29m1.597260032s) [noise=true , address=145.79.227.35:4497 , piece_id=YDNRFAJVXHRJD5QHICHR6CGF6K5ARZTLHOUSRLV6JAYU3ZSWEY3A, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.726843697 (640h29m1.597926742s) [noise=true , address=77.90.31.19:4846 , piece_id=2XJSN2WTR2BAVKDJZXEMY6SRX7KLOU4SVGYH7TNJIJTPW723DEHQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.727470170 (640h29m1.598553225s) [noise=true , address=145.79.227.30:4784 , piece_id=HA6Z6M3HASDLIDP2PHS7MUDOHBF6T6ZNPTABXF5PYXR2HFYSBOPQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.728088075 (640h29m1.59917112s) [noise=true , address=145.79.227.39:4032 , piece_id=Z7BQPDXOVQ6AEKACBB2DJFI652AT6UIHXUB2HPUJFMK53EF73AWQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:03.152355605 (640h29m3.02343864s) [upload=561203]: Next returned nil segment
Jan 1 07:58:03.152371765 (640h29m3.023454811s) [upload=561203]: Waiting for error group managing segments...
Jan 1 07:58:03.563991563 (640h29m3.435074608s) [noise=true , address=145.79.227.9:4391 , piece_id=XS6HYQTJO355YLXEIDBKHTJ44UAT5UBKHDUCFA744X6TBPGYN5YQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:03.652285026 (640h29m3.523368071s) [noise=true , address=145.79.227.49:4371 , piece_id=RGZZMDFFVVSIQJ2ZEUSK275SBWMGV75WGNDLY6LJ2TNRPUSY7RMQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:03.671359306 (640h29m3.542442351s) [noise=true , address=145.79.227.39:4032 , piece_id=Z7BQPDXOVQ6AEKACBB2DJFI652AT6UIHXUB2HPUJFMK53EF73AWQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:03.774991565 (640h29m3.64607462s) [noise=true , address=77.90.31.19:4846 , piece_id=2XJSN2WTR2BAVKDJZXEMY6SRX7KLOU4SVGYH7TNJIJTPW723DEHQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:03.870781221 (640h29m3.741864266s) [noise=true , address=77.90.31.12:4606 , piece_id=K4E5ECNMOMKEIFHZ3EZ2QSQHZ5IYP3CAD5DA6DBQLM4JYMOYU7MA, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:03.878179488 (640h29m3.749262533s) [noise=true , address=77.90.31.13:4649 , piece_id=YRKZDAT5O2USQM4SO5GRZ45SX7MA3U52KN2CRGYPX5WJ4HIKOJBA, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:03.977331467 (640h29m3.848414512s) [noise=true , address=77.90.31.15:4725 , piece_id=LLGZBPL7CHDI2TAY2SYMUMC264USSUQRVFR5P4HTLBWPBOTSUSOA, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.224768738 (640h29m4.095851783s) [noise=true , address=45.91.116.12:4238 , piece_id=R5LPEYTYC3XB76SSMDVHPZORAWB2P3S4TETMYV23NJI4UXYUYCEQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.457462948 (640h29m4.328545994s) [noise=true , address=45.91.116.5:4320 , piece_id=FOFLP7277VUUPTRZZHQLNVJ3YVYBELJTUYL7PTT7FJNN7J7QLS2A, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.488291199 (640h29m4.359374234s) [noise=true , address=145.79.227.35:4497 , piece_id=YDNRFAJVXHRJD5QHICHR6CGF6K5ARZTLHOUSRLV6JAYU3ZSWEY3A, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.522541837 (640h29m4.393624882s) [noise=true , address=145.79.227.43:4171 , piece_id=KYGE2RFI6O6DUFWVGJV2IT5VE3U43WJ4ODNQMTZFQH2DAADFC6BQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.525844266 (640h29m4.396927321s) [noise=true , address=145.79.227.33:4886 , piece_id=CYGCX4TM2ZFPCJYVDF52PDNR2OCNZ2NUSUUOUMUHYGCADWWFDHVQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.539045983 (640h29m4.410129028s) [noise=true , address=145.79.227.34:4924 , piece_id=QUPSGORFPMQVD2IRAK7COO2EE7XDV2KWEBTI2FQ2C25RTAAFNSRA, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.544558039 (640h29m4.415641084s) [noise=true , address=145.79.227.30:4784 , piece_id=HA6Z6M3HASDLIDP2PHS7MUDOHBF6T6ZNPTABXF5PYXR2HFYSBOPQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.544582956 (640h29m4.415665991s) [seg_pos={0 0}, upload=561203]: Segment reached optimal threshold of 14 pieces.
Jan 1 07:58:04.544811462 (640h29m4.415894527s) [noise=true , address=193.149.164.143:4694 , piece_id=EUO4UWMROIFFAJECCFRJBQF4XV25VDKGKX3UF2PTUMTE5JBGD67Q, seg_pos={0 0}, upload=561203]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx): protocol: expected piece hash; context canceled; EOF
Jan 1 07:58:04.544821240 (640h29m4.415904275s) [noise=true , address=193.149.164.143:4694 , piece_id=EUO4UWMROIFFAJECCFRJBQF4XV25VDKGKX3UF2PTUMTE5JBGD67Q, seg_pos={0 0}, upload=561203]: Long tail cancellation due to optimal threshold
Jan 1 07:58:04.545116732 (640h29m4.416199767s) [noise=true , address=64.49.13.169:4705 , piece_id=2P3JANSNK64726X7IYJOIQOKAQ4P4LJ5A2GVPX6UXOWUDIYRPZMA, seg_pos={0 0}, upload=561203]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:45312->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:45312->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45312->64.49.13.169:4705: use of closed network connection
Jan 1 07:58:04.545123666 (640h29m4.416206711s) [noise=true , address=64.49.13.169:4705 , piece_id=2P3JANSNK64726X7IYJOIQOKAQ4P4LJ5A2GVPX6UXOWUDIYRPZMA, seg_pos={0 0}, upload=561203]: Long tail cancellation due to optimal threshold
Jan 1 07:58:04.545303088 (640h29m4.416386123s) [noise=true , address=145.79.227.40:4066 , piece_id=ABZ7356MEBO4XPYAKFSS4EQJYJN4CWNJR3C6NXFWLAEO65KL3J5Q, seg_pos={0 0}, upload=561203]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF
Jan 1 07:58:04.545312055 (640h29m4.4163951s) [noise=true , address=145.79.227.40:4066 , piece_id=ABZ7356MEBO4XPYAKFSS4EQJYJN4CWNJR3C6NXFWLAEO65KL3J5Q, seg_pos={0 0}, upload=561203]: Long tail cancellation due to optimal threshold
Jan 1 07:58:04.545414279 (640h29m4.416497314s) [noise=true , address=64.49.13.137:4494 , piece_id=GCFOH6XRKDEGNG6TFPCMHG5RUCP2LUFS3XEOPNF42VYTGS63ZHQQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TX66fDVv3qu4yBGx9FNZTiArNeUnqxjMYwb3VGqYCQRe8RwoU): protocol: write tcp 188.166.225.170:35392->64.49.13.137:4494: use of closed network connection; write tcp 188.166.225.170:35392->64.49.13.137:4494: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35392->64.49.13.137:4494: use of closed network connection
Jan 1 07:58:04.545422815 (640h29m4.41650585s) [noise=true , address=64.49.13.137:4494 , piece_id=GCFOH6XRKDEGNG6TFPCMHG5RUCP2LUFS3XEOPNF42VYTGS63ZHQQ, seg_pos={0 0}, upload=561203]: Long tail cancellation due to optimal threshold
Jan 1 07:58:04.545439939 (640h29m4.416522974s) [seg_pos={0 0}, upload=561203]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 1 07:58:04.545580406 (640h29m4.416663441s) [upload=561203]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 1 07:58:04.545587800 (640h29m4.416670845s) [upload=561203]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 1 07:58:04.840726371 (640h29m4.711809416s) [upload=561203]: Done uploading segments...
Jan 1 07:58:04.840749646 (640h29m4.711832691s) [upload=561203]: Upload finished. err: <nil>
Duration: 36.660597068s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [X.X.XX.XX..X...X.XX.X...X.........X.X........XX.X...X..X....] duration:
6.023141091s trace:
https://jaeger.dev.storj.io/trace/14a6f6c583f5aa92
FAIL (2026-01-01 08:03:36 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: BPDHJMIUVCABD22C32WNGZZLNGGMWHI46PMQ7MBXBEDYPMTYPRZQ) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: ZN4WX6RP777Z6PHCE6UZOKKCK72YOXTCIHBSGNCWGMVSCC3ZW4KA) context deadline exceeded
======LOGS======
Jan 1 08:00:36.336451400 (640h31m36.207534445s) [upload=561245]: Starting upload
Jan 1 08:00:36.336534237 (640h31m36.207617283s) [upload=561245]: Done starting upload
Jan 1 08:00:36.336736032 (640h31m36.207819077s) [upload=561245]: Uploading segments...
Jan 1 08:00:36.336819661 (640h31m36.207902706s) [upload=561245]: Got next segment. Inline: false
Jan 1 08:00:36.336868724 (640h31m36.207951759s) [upload=561245]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 1 08:00:36.336875117 (640h31m36.207958162s) [upload=561245]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 1 08:00:36.580601654 (640h31m36.451684689s) [seg_pos={0 0}, upload=561245]: Begin upload segment...
Jan 1 08:00:36.580652981 (640h31m36.451736026s) [seg_pos={0 0}, upload=561245]: Done begin upload segment.
Jan 1 08:00:36.580707806 (640h31m36.451790851s) [noise=true , address=77.90.31.19:4841 , piece_id=W4EMUECOZ6NXFBZROVL6W6SI2AADAT6QZMKVZWFUOOJ6RPUHOGQQ, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.580843394 (640h31m36.451926429s) [noise=true , address=145.79.227.21:4475 , piece_id=H6RSWRNNW7U5XNKC27OEKZ5ILDOX73OCOZ6N2L35RKZ6TSZPHNFQ, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.581549899 (640h31m36.452632944s) [noise=true , address=145.79.227.29:4942 , piece_id=WDSA5DJWK3OVIAOAUCR4I2VTSG4HD4UT7EMV7SECRQHWUPZV2Y4A, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.582165562 (640h31m36.453248597s) [noise=true , address=145.79.227.48:4302 , piece_id=NJ3UEHC2WBCH7KJO3WZVHXKP3FDLHXKQULVEL7YMNMBU4OIKZUOA, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.582576053 (640h31m36.453659088s) [noise=true , address=145.79.227.30:4762 , piece_id=WZI2E3HGEQXIMFQBWBZ7PK2W5NDZMNBCKRF3EDB4MSOCA3DGD7JQ, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.583033354 (640h31m36.454116399s) [noise=true , address=145.79.227.18:4007 , piece_id=G4NXSCJCZ6LN2RLQSFGQX4XKLNPZFFZUOFROM3SFMOGFUYLWRHNQ, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.583471328 (640h31m36.454554373s) [noise=true , address=45.91.116.5:4297 , piece_id=XEGWKAMHMY3CRSMXGNAL4CDFDEFIVEDQINC7TXKILYIV73QV4UZQ, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.584935826 (640h31m36.456018871s) [noise=true , address=77.90.31.14:4661 , piece_id=BPDHJMIUVCABD22C32WNGZZLNGGMWHI46PMQ7MBXBEDYPMTYPRZQ, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.585046487 (640h31m36.456129522s) [noise=true , address=145.79.227.14:4840 , piece_id=54YIHG6H34PFOQO4OU3MV74CZUCFY5N2OCATSIOCMD453XOUANWQ, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.586542315 (640h31m36.45762535s) [noise=true , address=145.79.227.45:4230 , piece_id=6S3NFOZFIRQUE3T3PEZLZLMNVIOK33JUSYVH5HVWEJ3RMSDKAFXQ, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.586809845 (640h31m36.45789288s) [noise=true , address=145.79.227.33:4870 , piece_id=OYMN4GE2TWKCSLFDJ66GJ6UB2ZXDHK7D4CC5A7AYH4NGJCAT66JA, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.588268963 (640h31m36.459351998s) [noise=true , address=145.79.227.24:4581 , piece_id=DRKKS2N35E7ORYSCARSNHBNKN3DTIGV5STIISCK7THZW76CDVO3A, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.590930822 (640h31m36.462013867s) [noise=true , address=145.79.227.38:4977 , piece_id=J3YUHNPAJEX5D76LVFHZBAP73HSG5IN7ZI5VD7IX4ZVMVGHSAHAA, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.594801081 (640h31m36.465884116s) [noise=true , address=145.79.227.52:4447 , piece_id=R7LXPRI2SWX2PAAALMPFOHP3JNJAFJ5QLJINAFKVK7H6PEE2TTGA, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.598974989 (640h31m36.470058024s) [noise=true , address=145.79.227.9:4367 , piece_id=JGIZMNV72PTWNVAKBO2CQMG6VGXJA6YUJABLSTORA3ZMDO6HYHRQ, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.600900144 (640h31m36.47198319s) [noise=true , address=45.91.116.8:4048 , piece_id=ZN4WX6RP777Z6PHCE6UZOKKCK72YOXTCIHBSGNCWGMVSCC3ZW4KA, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:38.299065741 (640h31m38.170148776s) [upload=561245]: Next returned nil segment
Jan 1 08:00:38.299077214 (640h31m38.170160259s) [upload=561245]: Waiting for error group managing segments...
Jan 1 08:00:38.586542564 (640h31m38.45762562s) [noise=true , address=145.79.227.21:4475 , piece_id=H6RSWRNNW7U5XNKC27OEKZ5ILDOX73OCOZ6N2L35RKZ6TSZPHNFQ, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:38.587267896 (640h31m38.458350941s) [noise=true , address=145.79.227.52:4447 , piece_id=R7LXPRI2SWX2PAAALMPFOHP3JNJAFJ5QLJINAFKVK7H6PEE2TTGA, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:38.587822492 (640h31m38.458905537s) [noise=true , address=145.79.227.18:4007 , piece_id=G4NXSCJCZ6LN2RLQSFGQX4XKLNPZFFZUOFROM3SFMOGFUYLWRHNQ, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:38.600692919 (640h31m38.471775964s) [noise=true , address=145.79.227.45:4230 , piece_id=6S3NFOZFIRQUE3T3PEZLZLMNVIOK33JUSYVH5HVWEJ3RMSDKAFXQ, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:38.601771583 (640h31m38.472854628s) [noise=true , address=145.79.227.48:4302 , piece_id=NJ3UEHC2WBCH7KJO3WZVHXKP3FDLHXKQULVEL7YMNMBU4OIKZUOA, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:38.606091358 (640h31m38.477174403s) [noise=true , address=145.79.227.38:4977 , piece_id=J3YUHNPAJEX5D76LVFHZBAP73HSG5IN7ZI5VD7IX4ZVMVGHSAHAA, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:38.611687373 (640h31m38.482770418s) [noise=true , address=145.79.227.14:4840 , piece_id=54YIHG6H34PFOQO4OU3MV74CZUCFY5N2OCATSIOCMD453XOUANWQ, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:38.614887266 (640h31m38.485970312s) [noise=true , address=145.79.227.24:4581 , piece_id=DRKKS2N35E7ORYSCARSNHBNKN3DTIGV5STIISCK7THZW76CDVO3A, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:38.626493386 (640h31m38.497576431s) [noise=true , address=145.79.227.9:4367 , piece_id=JGIZMNV72PTWNVAKBO2CQMG6VGXJA6YUJABLSTORA3ZMDO6HYHRQ, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:38.910948826 (640h31m38.782031871s) [noise=true , address=45.91.116.5:4297 , piece_id=XEGWKAMHMY3CRSMXGNAL4CDFDEFIVEDQINC7TXKILYIV73QV4UZQ, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:38.912793028 (640h31m38.783876063s) [noise=true , address=77.90.31.14:4661 , piece_id=BPDHJMIUVCABD22C32WNGZZLNGGMWHI46PMQ7MBXBEDYPMTYPRZQ, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:38.942787882 (640h31m38.813870917s) [noise=true , address=77.90.31.19:4841 , piece_id=W4EMUECOZ6NXFBZROVL6W6SI2AADAT6QZMKVZWFUOOJ6RPUHOGQQ, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:39.097452903 (640h31m38.968535948s) [noise=true , address=45.91.116.8:4048 , piece_id=ZN4WX6RP777Z6PHCE6UZOKKCK72YOXTCIHBSGNCWGMVSCC3ZW4KA, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:39.097478132 (640h31m38.968561167s) [seg_pos={0 0}, upload=561245]: Segment reached optimal threshold of 13 pieces.
Jan 1 08:00:39.097913300 (640h31m38.968996335s) [noise=true , address=145.79.227.33:4870 , piece_id=OYMN4GE2TWKCSLFDJ66GJ6UB2ZXDHK7D4CC5A7AYH4NGJCAT66JA, seg_pos={0 0}, upload=561245]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF
Jan 1 08:00:39.097923670 (640h31m38.969006705s) [noise=true , address=145.79.227.33:4870 , piece_id=OYMN4GE2TWKCSLFDJ66GJ6UB2ZXDHK7D4CC5A7AYH4NGJCAT66JA, seg_pos={0 0}, upload=561245]: Long tail cancellation due to optimal threshold
Jan 1 08:00:39.097970158 (640h31m38.969053203s) [noise=true , address=145.79.227.30:4762 , piece_id=WZI2E3HGEQXIMFQBWBZ7PK2W5NDZMNBCKRF3EDB4MSOCA3DGD7JQ, seg_pos={0 0}, upload=561245]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF
Jan 1 08:00:39.097981329 (640h31m38.969064374s) [noise=true , address=145.79.227.30:4762 , piece_id=WZI2E3HGEQXIMFQBWBZ7PK2W5NDZMNBCKRF3EDB4MSOCA3DGD7JQ, seg_pos={0 0}, upload=561245]: Long tail cancellation due to optimal threshold
Jan 1 08:00:39.098151002 (640h31m38.969234048s) [noise=true , address=145.79.227.29:4942 , piece_id=WDSA5DJWK3OVIAOAUCR4I2VTSG4HD4UT7EMV7SECRQHWUPZV2Y4A, seg_pos={0 0}, upload=561245]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF
Jan 1 08:00:39.098160401 (640h31m38.969243436s) [noise=true , address=145.79.227.29:4942 , piece_id=WDSA5DJWK3OVIAOAUCR4I2VTSG4HD4UT7EMV7SECRQHWUPZV2Y4A, seg_pos={0 0}, upload=561245]: Long tail cancellation due to optimal threshold
Jan 1 08:00:39.098182162 (640h31m38.969265207s) [seg_pos={0 0}, upload=561245]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 1 08:00:39.098277884 (640h31m38.96936094s) [upload=561245]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 1 08:00:39.098285188 (640h31m38.969368243s) [upload=561245]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 1 08:00:39.388401257 (640h31m39.259484302s) [upload=561245]: Done uploading segments...
Jan 1 08:00:39.388428308 (640h31m39.259511343s) [upload=561245]: Upload finished. err: <nil>
Duration: 6.02317688s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [.X.X.XX.XX..X...X.XX.X...X.........X.X........XX.X...X..X...] duration:
6.19570865s trace:
https://jaeger.dev.storj.io/trace/14a6f6c583f5aa92
FAIL (2026-01-01 08:03:20 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: BPDHJMIUVCABD22C32WNGZZLNGGMWHI46PMQ7MBXBEDYPMTYPRZQ) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: ZN4WX6RP777Z6PHCE6UZOKKCK72YOXTCIHBSGNCWGMVSCC3ZW4KA) context deadline exceeded
======LOGS======
Jan 1 08:00:36.336451400 (640h31m36.207534445s) [upload=561245]: Starting upload
Jan 1 08:00:36.336534237 (640h31m36.207617283s) [upload=561245]: Done starting upload
Jan 1 08:00:36.336736032 (640h31m36.207819077s) [upload=561245]: Uploading segments...
Jan 1 08:00:36.336819661 (640h31m36.207902706s) [upload=561245]: Got next segment. Inline: false
Jan 1 08:00:36.336868724 (640h31m36.207951759s) [upload=561245]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 1 08:00:36.336875117 (640h31m36.207958162s) [upload=561245]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 1 08:00:36.580601654 (640h31m36.451684689s) [seg_pos={0 0}, upload=561245]: Begin upload segment...
Jan 1 08:00:36.580652981 (640h31m36.451736026s) [seg_pos={0 0}, upload=561245]: Done begin upload segment.
Jan 1 08:00:36.580707806 (640h31m36.451790851s) [noise=true , address=77.90.31.19:4841 , piece_id=W4EMUECOZ6NXFBZROVL6W6SI2AADAT6QZMKVZWFUOOJ6RPUHOGQQ, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.580843394 (640h31m36.451926429s) [noise=true , address=145.79.227.21:4475 , piece_id=H6RSWRNNW7U5XNKC27OEKZ5ILDOX73OCOZ6N2L35RKZ6TSZPHNFQ, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.581549899 (640h31m36.452632944s) [noise=true , address=145.79.227.29:4942 , piece_id=WDSA5DJWK3OVIAOAUCR4I2VTSG4HD4UT7EMV7SECRQHWUPZV2Y4A, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.582165562 (640h31m36.453248597s) [noise=true , address=145.79.227.48:4302 , piece_id=NJ3UEHC2WBCH7KJO3WZVHXKP3FDLHXKQULVEL7YMNMBU4OIKZUOA, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.582576053 (640h31m36.453659088s) [noise=true , address=145.79.227.30:4762 , piece_id=WZI2E3HGEQXIMFQBWBZ7PK2W5NDZMNBCKRF3EDB4MSOCA3DGD7JQ, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.583033354 (640h31m36.454116399s) [noise=true , address=145.79.227.18:4007 , piece_id=G4NXSCJCZ6LN2RLQSFGQX4XKLNPZFFZUOFROM3SFMOGFUYLWRHNQ, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.583471328 (640h31m36.454554373s) [noise=true , address=45.91.116.5:4297 , piece_id=XEGWKAMHMY3CRSMXGNAL4CDFDEFIVEDQINC7TXKILYIV73QV4UZQ, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.584935826 (640h31m36.456018871s) [noise=true , address=77.90.31.14:4661 , piece_id=BPDHJMIUVCABD22C32WNGZZLNGGMWHI46PMQ7MBXBEDYPMTYPRZQ, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.585046487 (640h31m36.456129522s) [noise=true , address=145.79.227.14:4840 , piece_id=54YIHG6H34PFOQO4OU3MV74CZUCFY5N2OCATSIOCMD453XOUANWQ, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.586542315 (640h31m36.45762535s) [noise=true , address=145.79.227.45:4230 , piece_id=6S3NFOZFIRQUE3T3PEZLZLMNVIOK33JUSYVH5HVWEJ3RMSDKAFXQ, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.586809845 (640h31m36.45789288s) [noise=true , address=145.79.227.33:4870 , piece_id=OYMN4GE2TWKCSLFDJ66GJ6UB2ZXDHK7D4CC5A7AYH4NGJCAT66JA, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.588268963 (640h31m36.459351998s) [noise=true , address=145.79.227.24:4581 , piece_id=DRKKS2N35E7ORYSCARSNHBNKN3DTIGV5STIISCK7THZW76CDVO3A, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.590930822 (640h31m36.462013867s) [noise=true , address=145.79.227.38:4977 , piece_id=J3YUHNPAJEX5D76LVFHZBAP73HSG5IN7ZI5VD7IX4ZVMVGHSAHAA, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.594801081 (640h31m36.465884116s) [noise=true , address=145.79.227.52:4447 , piece_id=R7LXPRI2SWX2PAAALMPFOHP3JNJAFJ5QLJINAFKVK7H6PEE2TTGA, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.598974989 (640h31m36.470058024s) [noise=true , address=145.79.227.9:4367 , piece_id=JGIZMNV72PTWNVAKBO2CQMG6VGXJA6YUJABLSTORA3ZMDO6HYHRQ, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.600900144 (640h31m36.47198319s) [noise=true , address=45.91.116.8:4048 , piece_id=ZN4WX6RP777Z6PHCE6UZOKKCK72YOXTCIHBSGNCWGMVSCC3ZW4KA, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:38.299065741 (640h31m38.170148776s) [upload=561245]: Next returned nil segment
Jan 1 08:00:38.299077214 (640h31m38.170160259s) [upload=561245]: Waiting for error group managing segments...
Jan 1 08:00:38.586542564 (640h31m38.45762562s) [noise=true , address=145.79.227.21:4475 , piece_id=H6RSWRNNW7U5XNKC27OEKZ5ILDOX73OCOZ6N2L35RKZ6TSZPHNFQ, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:38.587267896 (640h31m38.458350941s) [noise=true , address=145.79.227.52:4447 , piece_id=R7LXPRI2SWX2PAAALMPFOHP3JNJAFJ5QLJINAFKVK7H6PEE2TTGA, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:38.587822492 (640h31m38.458905537s) [noise=true , address=145.79.227.18:4007 , piece_id=G4NXSCJCZ6LN2RLQSFGQX4XKLNPZFFZUOFROM3SFMOGFUYLWRHNQ, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:38.600692919 (640h31m38.471775964s) [noise=true , address=145.79.227.45:4230 , piece_id=6S3NFOZFIRQUE3T3PEZLZLMNVIOK33JUSYVH5HVWEJ3RMSDKAFXQ, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:38.601771583 (640h31m38.472854628s) [noise=true , address=145.79.227.48:4302 , piece_id=NJ3UEHC2WBCH7KJO3WZVHXKP3FDLHXKQULVEL7YMNMBU4OIKZUOA, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:38.606091358 (640h31m38.477174403s) [noise=true , address=145.79.227.38:4977 , piece_id=J3YUHNPAJEX5D76LVFHZBAP73HSG5IN7ZI5VD7IX4ZVMVGHSAHAA, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:38.611687373 (640h31m38.482770418s) [noise=true , address=145.79.227.14:4840 , piece_id=54YIHG6H34PFOQO4OU3MV74CZUCFY5N2OCATSIOCMD453XOUANWQ, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:38.614887266 (640h31m38.485970312s) [noise=true , address=145.79.227.24:4581 , piece_id=DRKKS2N35E7ORYSCARSNHBNKN3DTIGV5STIISCK7THZW76CDVO3A, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:38.626493386 (640h31m38.497576431s) [noise=true , address=145.79.227.9:4367 , piece_id=JGIZMNV72PTWNVAKBO2CQMG6VGXJA6YUJABLSTORA3ZMDO6HYHRQ, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:38.910948826 (640h31m38.782031871s) [noise=true , address=45.91.116.5:4297 , piece_id=XEGWKAMHMY3CRSMXGNAL4CDFDEFIVEDQINC7TXKILYIV73QV4UZQ, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:38.912793028 (640h31m38.783876063s) [noise=true , address=77.90.31.14:4661 , piece_id=BPDHJMIUVCABD22C32WNGZZLNGGMWHI46PMQ7MBXBEDYPMTYPRZQ, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:38.942787882 (640h31m38.813870917s) [noise=true , address=77.90.31.19:4841 , piece_id=W4EMUECOZ6NXFBZROVL6W6SI2AADAT6QZMKVZWFUOOJ6RPUHOGQQ, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:39.097452903 (640h31m38.968535948s) [noise=true , address=45.91.116.8:4048 , piece_id=ZN4WX6RP777Z6PHCE6UZOKKCK72YOXTCIHBSGNCWGMVSCC3ZW4KA, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:39.097478132 (640h31m38.968561167s) [seg_pos={0 0}, upload=561245]: Segment reached optimal threshold of 13 pieces.
Jan 1 08:00:39.097913300 (640h31m38.968996335s) [noise=true , address=145.79.227.33:4870 , piece_id=OYMN4GE2TWKCSLFDJ66GJ6UB2ZXDHK7D4CC5A7AYH4NGJCAT66JA, seg_pos={0 0}, upload=561245]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF
Jan 1 08:00:39.097923670 (640h31m38.969006705s) [noise=true , address=145.79.227.33:4870 , piece_id=OYMN4GE2TWKCSLFDJ66GJ6UB2ZXDHK7D4CC5A7AYH4NGJCAT66JA, seg_pos={0 0}, upload=561245]: Long tail cancellation due to optimal threshold
Jan 1 08:00:39.097970158 (640h31m38.969053203s) [noise=true , address=145.79.227.30:4762 , piece_id=WZI2E3HGEQXIMFQBWBZ7PK2W5NDZMNBCKRF3EDB4MSOCA3DGD7JQ, seg_pos={0 0}, upload=561245]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF
Jan 1 08:00:39.097981329 (640h31m38.969064374s) [noise=true , address=145.79.227.30:4762 , piece_id=WZI2E3HGEQXIMFQBWBZ7PK2W5NDZMNBCKRF3EDB4MSOCA3DGD7JQ, seg_pos={0 0}, upload=561245]: Long tail cancellation due to optimal threshold
Jan 1 08:00:39.098151002 (640h31m38.969234048s) [noise=true , address=145.79.227.29:4942 , piece_id=WDSA5DJWK3OVIAOAUCR4I2VTSG4HD4UT7EMV7SECRQHWUPZV2Y4A, seg_pos={0 0}, upload=561245]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF
Jan 1 08:00:39.098160401 (640h31m38.969243436s) [noise=true , address=145.79.227.29:4942 , piece_id=WDSA5DJWK3OVIAOAUCR4I2VTSG4HD4UT7EMV7SECRQHWUPZV2Y4A, seg_pos={0 0}, upload=561245]: Long tail cancellation due to optimal threshold
Jan 1 08:00:39.098182162 (640h31m38.969265207s) [seg_pos={0 0}, upload=561245]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 1 08:00:39.098277884 (640h31m38.96936094s) [upload=561245]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 1 08:00:39.098285188 (640h31m38.969368243s) [upload=561245]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 1 08:00:39.388401257 (640h31m39.259484302s) [upload=561245]: Done uploading segments...
Jan 1 08:00:39.388428308 (640h31m39.259511343s) [upload=561245]: Upload finished. err: <nil>
Duration: 6.195742966s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
38 out of 60 is failed (threshold=6) [XX.XX.X.XXXXX.XX..XXXXX.XXXX..XX.XXX.XX...X.X..X...XXXXXX..X] duration:
5.001562281s trace:
https://jaeger.dev.storj.io/trace/d9572936aa84902
FAIL (2026-01-01 08:03:19 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001582409s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
18.953856454s trace:
https://jaeger.dev.storj.io/trace/44e9d74e3b154e8e
FAIL (2026-01-01 08:03:09 UTC)
orbiter warning: context deadline exceeded
Duration: 18.953883584s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
37 out of 60 is failed (threshold=6) [.XX.XX.X.XXXXX.XX..XXXXX.XXXX..XX.XXX.XX...X.X..X...XXXXXX..] duration:
3.673099455s trace:
https://jaeger.dev.storj.io/trace/5195a3bdc9c8f86b
FAIL (2026-01-01 08:02:44 UTC)
orbiter warning: context deadline exceeded
Duration: 3.673129744s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [..X.X.XX.XX..X...X.XX.X...X.........X.X........XX.X...X..X..] duration:
28.068027166s trace:
https://jaeger.dev.storj.io/trace/14a6f6c583f5aa92
FAIL (2026-01-01 08:02:37 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: BPDHJMIUVCABD22C32WNGZZLNGGMWHI46PMQ7MBXBEDYPMTYPRZQ) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: ZN4WX6RP777Z6PHCE6UZOKKCK72YOXTCIHBSGNCWGMVSCC3ZW4KA) context deadline exceeded
======LOGS======
Jan 1 08:00:36.336451400 (640h31m36.207534445s) [upload=561245]: Starting upload
Jan 1 08:00:36.336534237 (640h31m36.207617283s) [upload=561245]: Done starting upload
Jan 1 08:00:36.336736032 (640h31m36.207819077s) [upload=561245]: Uploading segments...
Jan 1 08:00:36.336819661 (640h31m36.207902706s) [upload=561245]: Got next segment. Inline: false
Jan 1 08:00:36.336868724 (640h31m36.207951759s) [upload=561245]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 1 08:00:36.336875117 (640h31m36.207958162s) [upload=561245]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 1 08:00:36.580601654 (640h31m36.451684689s) [seg_pos={0 0}, upload=561245]: Begin upload segment...
Jan 1 08:00:36.580652981 (640h31m36.451736026s) [seg_pos={0 0}, upload=561245]: Done begin upload segment.
Jan 1 08:00:36.580707806 (640h31m36.451790851s) [noise=true , address=77.90.31.19:4841 , piece_id=W4EMUECOZ6NXFBZROVL6W6SI2AADAT6QZMKVZWFUOOJ6RPUHOGQQ, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.580843394 (640h31m36.451926429s) [noise=true , address=145.79.227.21:4475 , piece_id=H6RSWRNNW7U5XNKC27OEKZ5ILDOX73OCOZ6N2L35RKZ6TSZPHNFQ, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.581549899 (640h31m36.452632944s) [noise=true , address=145.79.227.29:4942 , piece_id=WDSA5DJWK3OVIAOAUCR4I2VTSG4HD4UT7EMV7SECRQHWUPZV2Y4A, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.582165562 (640h31m36.453248597s) [noise=true , address=145.79.227.48:4302 , piece_id=NJ3UEHC2WBCH7KJO3WZVHXKP3FDLHXKQULVEL7YMNMBU4OIKZUOA, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.582576053 (640h31m36.453659088s) [noise=true , address=145.79.227.30:4762 , piece_id=WZI2E3HGEQXIMFQBWBZ7PK2W5NDZMNBCKRF3EDB4MSOCA3DGD7JQ, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.583033354 (640h31m36.454116399s) [noise=true , address=145.79.227.18:4007 , piece_id=G4NXSCJCZ6LN2RLQSFGQX4XKLNPZFFZUOFROM3SFMOGFUYLWRHNQ, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.583471328 (640h31m36.454554373s) [noise=true , address=45.91.116.5:4297 , piece_id=XEGWKAMHMY3CRSMXGNAL4CDFDEFIVEDQINC7TXKILYIV73QV4UZQ, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.584935826 (640h31m36.456018871s) [noise=true , address=77.90.31.14:4661 , piece_id=BPDHJMIUVCABD22C32WNGZZLNGGMWHI46PMQ7MBXBEDYPMTYPRZQ, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.585046487 (640h31m36.456129522s) [noise=true , address=145.79.227.14:4840 , piece_id=54YIHG6H34PFOQO4OU3MV74CZUCFY5N2OCATSIOCMD453XOUANWQ, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.586542315 (640h31m36.45762535s) [noise=true , address=145.79.227.45:4230 , piece_id=6S3NFOZFIRQUE3T3PEZLZLMNVIOK33JUSYVH5HVWEJ3RMSDKAFXQ, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.586809845 (640h31m36.45789288s) [noise=true , address=145.79.227.33:4870 , piece_id=OYMN4GE2TWKCSLFDJ66GJ6UB2ZXDHK7D4CC5A7AYH4NGJCAT66JA, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.588268963 (640h31m36.459351998s) [noise=true , address=145.79.227.24:4581 , piece_id=DRKKS2N35E7ORYSCARSNHBNKN3DTIGV5STIISCK7THZW76CDVO3A, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.590930822 (640h31m36.462013867s) [noise=true , address=145.79.227.38:4977 , piece_id=J3YUHNPAJEX5D76LVFHZBAP73HSG5IN7ZI5VD7IX4ZVMVGHSAHAA, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.594801081 (640h31m36.465884116s) [noise=true , address=145.79.227.52:4447 , piece_id=R7LXPRI2SWX2PAAALMPFOHP3JNJAFJ5QLJINAFKVK7H6PEE2TTGA, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.598974989 (640h31m36.470058024s) [noise=true , address=145.79.227.9:4367 , piece_id=JGIZMNV72PTWNVAKBO2CQMG6VGXJA6YUJABLSTORA3ZMDO6HYHRQ, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.600900144 (640h31m36.47198319s) [noise=true , address=45.91.116.8:4048 , piece_id=ZN4WX6RP777Z6PHCE6UZOKKCK72YOXTCIHBSGNCWGMVSCC3ZW4KA, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:38.299065741 (640h31m38.170148776s) [upload=561245]: Next returned nil segment
Jan 1 08:00:38.299077214 (640h31m38.170160259s) [upload=561245]: Waiting for error group managing segments...
Jan 1 08:00:38.586542564 (640h31m38.45762562s) [noise=true , address=145.79.227.21:4475 , piece_id=H6RSWRNNW7U5XNKC27OEKZ5ILDOX73OCOZ6N2L35RKZ6TSZPHNFQ, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:38.587267896 (640h31m38.458350941s) [noise=true , address=145.79.227.52:4447 , piece_id=R7LXPRI2SWX2PAAALMPFOHP3JNJAFJ5QLJINAFKVK7H6PEE2TTGA, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:38.587822492 (640h31m38.458905537s) [noise=true , address=145.79.227.18:4007 , piece_id=G4NXSCJCZ6LN2RLQSFGQX4XKLNPZFFZUOFROM3SFMOGFUYLWRHNQ, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:38.600692919 (640h31m38.471775964s) [noise=true , address=145.79.227.45:4230 , piece_id=6S3NFOZFIRQUE3T3PEZLZLMNVIOK33JUSYVH5HVWEJ3RMSDKAFXQ, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:38.601771583 (640h31m38.472854628s) [noise=true , address=145.79.227.48:4302 , piece_id=NJ3UEHC2WBCH7KJO3WZVHXKP3FDLHXKQULVEL7YMNMBU4OIKZUOA, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:38.606091358 (640h31m38.477174403s) [noise=true , address=145.79.227.38:4977 , piece_id=J3YUHNPAJEX5D76LVFHZBAP73HSG5IN7ZI5VD7IX4ZVMVGHSAHAA, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:38.611687373 (640h31m38.482770418s) [noise=true , address=145.79.227.14:4840 , piece_id=54YIHG6H34PFOQO4OU3MV74CZUCFY5N2OCATSIOCMD453XOUANWQ, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:38.614887266 (640h31m38.485970312s) [noise=true , address=145.79.227.24:4581 , piece_id=DRKKS2N35E7ORYSCARSNHBNKN3DTIGV5STIISCK7THZW76CDVO3A, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:38.626493386 (640h31m38.497576431s) [noise=true , address=145.79.227.9:4367 , piece_id=JGIZMNV72PTWNVAKBO2CQMG6VGXJA6YUJABLSTORA3ZMDO6HYHRQ, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:38.910948826 (640h31m38.782031871s) [noise=true , address=45.91.116.5:4297 , piece_id=XEGWKAMHMY3CRSMXGNAL4CDFDEFIVEDQINC7TXKILYIV73QV4UZQ, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:38.912793028 (640h31m38.783876063s) [noise=true , address=77.90.31.14:4661 , piece_id=BPDHJMIUVCABD22C32WNGZZLNGGMWHI46PMQ7MBXBEDYPMTYPRZQ, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:38.942787882 (640h31m38.813870917s) [noise=true , address=77.90.31.19:4841 , piece_id=W4EMUECOZ6NXFBZROVL6W6SI2AADAT6QZMKVZWFUOOJ6RPUHOGQQ, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:39.097452903 (640h31m38.968535948s) [noise=true , address=45.91.116.8:4048 , piece_id=ZN4WX6RP777Z6PHCE6UZOKKCK72YOXTCIHBSGNCWGMVSCC3ZW4KA, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:39.097478132 (640h31m38.968561167s) [seg_pos={0 0}, upload=561245]: Segment reached optimal threshold of 13 pieces.
Jan 1 08:00:39.097913300 (640h31m38.968996335s) [noise=true , address=145.79.227.33:4870 , piece_id=OYMN4GE2TWKCSLFDJ66GJ6UB2ZXDHK7D4CC5A7AYH4NGJCAT66JA, seg_pos={0 0}, upload=561245]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF
Jan 1 08:00:39.097923670 (640h31m38.969006705s) [noise=true , address=145.79.227.33:4870 , piece_id=OYMN4GE2TWKCSLFDJ66GJ6UB2ZXDHK7D4CC5A7AYH4NGJCAT66JA, seg_pos={0 0}, upload=561245]: Long tail cancellation due to optimal threshold
Jan 1 08:00:39.097970158 (640h31m38.969053203s) [noise=true , address=145.79.227.30:4762 , piece_id=WZI2E3HGEQXIMFQBWBZ7PK2W5NDZMNBCKRF3EDB4MSOCA3DGD7JQ, seg_pos={0 0}, upload=561245]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF
Jan 1 08:00:39.097981329 (640h31m38.969064374s) [noise=true , address=145.79.227.30:4762 , piece_id=WZI2E3HGEQXIMFQBWBZ7PK2W5NDZMNBCKRF3EDB4MSOCA3DGD7JQ, seg_pos={0 0}, upload=561245]: Long tail cancellation due to optimal threshold
Jan 1 08:00:39.098151002 (640h31m38.969234048s) [noise=true , address=145.79.227.29:4942 , piece_id=WDSA5DJWK3OVIAOAUCR4I2VTSG4HD4UT7EMV7SECRQHWUPZV2Y4A, seg_pos={0 0}, upload=561245]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF
Jan 1 08:00:39.098160401 (640h31m38.969243436s) [noise=true , address=145.79.227.29:4942 , piece_id=WDSA5DJWK3OVIAOAUCR4I2VTSG4HD4UT7EMV7SECRQHWUPZV2Y4A, seg_pos={0 0}, upload=561245]: Long tail cancellation due to optimal threshold
Jan 1 08:00:39.098182162 (640h31m38.969265207s) [seg_pos={0 0}, upload=561245]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 1 08:00:39.098277884 (640h31m38.96936094s) [upload=561245]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 1 08:00:39.098285188 (640h31m38.969368243s) [upload=561245]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 1 08:00:39.388401257 (640h31m39.259484302s) [upload=561245]: Done uploading segments...
Jan 1 08:00:39.388428308 (640h31m39.259511343s) [upload=561245]: Upload finished. err: <nil>
Duration: 28.068056582s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [................X.....X..XX.........................XX......] duration:
21.201987814s trace:
https://jaeger.dev.storj.io/trace/10d96081b66cd020
FAIL (2026-01-01 08:02:31 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
73329s) [noise=true , address=145.79.227.40:4066 , piece_id=ABZ7356MEBO4XPYAKFSS4EQJYJN4CWNJR3C6NXFWLAEO65KL3J5Q, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.722330803 (640h29m1.593413848s) [noise=true , address=145.79.227.9:4391 , piece_id=XS6HYQTJO355YLXEIDBKHTJ44UAT5UBKHDUCFA744X6TBPGYN5YQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.722848709 (640h29m1.593931754s) [noise=true , address=77.90.31.12:4606 , piece_id=K4E5ECNMOMKEIFHZ3EZ2QSQHZ5IYP3CAD5DA6DBQLM4JYMOYU7MA, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.723452529 (640h29m1.594535574s) [noise=true , address=193.149.164.143:4694 , piece_id=EUO4UWMROIFFAJECCFRJBQF4XV25VDKGKX3UF2PTUMTE5JBGD67Q, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.723730728 (640h29m1.594813773s) [noise=true , address=45.91.116.5:4320 , piece_id=FOFLP7277VUUPTRZZHQLNVJ3YVYBELJTUYL7PTT7FJNN7J7QLS2A, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.724307808 (640h29m1.595390853s) [noise=true , address=64.49.13.169:4705 , piece_id=2P3JANSNK64726X7IYJOIQOKAQ4P4LJ5A2GVPX6UXOWUDIYRPZMA, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.724847144 (640h29m1.595930189s) [noise=true , address=64.49.13.137:4494 , piece_id=GCFOH6XRKDEGNG6TFPCMHG5RUCP2LUFS3XEOPNF42VYTGS63ZHQQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.724970960 (640h29m1.596054005s) [noise=true , address=77.90.31.15:4725 , piece_id=LLGZBPL7CHDI2TAY2SYMUMC264USSUQRVFR5P4HTLBWPBOTSUSOA, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.725289036 (640h29m1.596372071s) [noise=true , address=145.79.227.49:4371 , piece_id=RGZZMDFFVVSIQJ2ZEUSK275SBWMGV75WGNDLY6LJ2TNRPUSY7RMQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.725518012 (640h29m1.596601047s) [noise=true , address=77.90.31.13:4649 , piece_id=YRKZDAT5O2USQM4SO5GRZ45SX7MA3U52KN2CRGYPX5WJ4HIKOJBA, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.725902264 (640h29m1.596985309s) [noise=true , address=45.91.116.12:4238 , piece_id=R5LPEYTYC3XB76SSMDVHPZORAWB2P3S4TETMYV23NJI4UXYUYCEQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.726176987 (640h29m1.597260032s) [noise=true , address=145.79.227.35:4497 , piece_id=YDNRFAJVXHRJD5QHICHR6CGF6K5ARZTLHOUSRLV6JAYU3ZSWEY3A, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.726843697 (640h29m1.597926742s) [noise=true , address=77.90.31.19:4846 , piece_id=2XJSN2WTR2BAVKDJZXEMY6SRX7KLOU4SVGYH7TNJIJTPW723DEHQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.727470170 (640h29m1.598553225s) [noise=true , address=145.79.227.30:4784 , piece_id=HA6Z6M3HASDLIDP2PHS7MUDOHBF6T6ZNPTABXF5PYXR2HFYSBOPQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.728088075 (640h29m1.59917112s) [noise=true , address=145.79.227.39:4032 , piece_id=Z7BQPDXOVQ6AEKACBB2DJFI652AT6UIHXUB2HPUJFMK53EF73AWQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:03.152355605 (640h29m3.02343864s) [upload=561203]: Next returned nil segment
Jan 1 07:58:03.152371765 (640h29m3.023454811s) [upload=561203]: Waiting for error group managing segments...
Jan 1 07:58:03.563991563 (640h29m3.435074608s) [noise=true , address=145.79.227.9:4391 , piece_id=XS6HYQTJO355YLXEIDBKHTJ44UAT5UBKHDUCFA744X6TBPGYN5YQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:03.652285026 (640h29m3.523368071s) [noise=true , address=145.79.227.49:4371 , piece_id=RGZZMDFFVVSIQJ2ZEUSK275SBWMGV75WGNDLY6LJ2TNRPUSY7RMQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:03.671359306 (640h29m3.542442351s) [noise=true , address=145.79.227.39:4032 , piece_id=Z7BQPDXOVQ6AEKACBB2DJFI652AT6UIHXUB2HPUJFMK53EF73AWQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:03.774991565 (640h29m3.64607462s) [noise=true , address=77.90.31.19:4846 , piece_id=2XJSN2WTR2BAVKDJZXEMY6SRX7KLOU4SVGYH7TNJIJTPW723DEHQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:03.870781221 (640h29m3.741864266s) [noise=true , address=77.90.31.12:4606 , piece_id=K4E5ECNMOMKEIFHZ3EZ2QSQHZ5IYP3CAD5DA6DBQLM4JYMOYU7MA, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:03.878179488 (640h29m3.749262533s) [noise=true , address=77.90.31.13:4649 , piece_id=YRKZDAT5O2USQM4SO5GRZ45SX7MA3U52KN2CRGYPX5WJ4HIKOJBA, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:03.977331467 (640h29m3.848414512s) [noise=true , address=77.90.31.15:4725 , piece_id=LLGZBPL7CHDI2TAY2SYMUMC264USSUQRVFR5P4HTLBWPBOTSUSOA, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.224768738 (640h29m4.095851783s) [noise=true , address=45.91.116.12:4238 , piece_id=R5LPEYTYC3XB76SSMDVHPZORAWB2P3S4TETMYV23NJI4UXYUYCEQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.457462948 (640h29m4.328545994s) [noise=true , address=45.91.116.5:4320 , piece_id=FOFLP7277VUUPTRZZHQLNVJ3YVYBELJTUYL7PTT7FJNN7J7QLS2A, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.488291199 (640h29m4.359374234s) [noise=true , address=145.79.227.35:4497 , piece_id=YDNRFAJVXHRJD5QHICHR6CGF6K5ARZTLHOUSRLV6JAYU3ZSWEY3A, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.522541837 (640h29m4.393624882s) [noise=true , address=145.79.227.43:4171 , piece_id=KYGE2RFI6O6DUFWVGJV2IT5VE3U43WJ4ODNQMTZFQH2DAADFC6BQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.525844266 (640h29m4.396927321s) [noise=true , address=145.79.227.33:4886 , piece_id=CYGCX4TM2ZFPCJYVDF52PDNR2OCNZ2NUSUUOUMUHYGCADWWFDHVQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.539045983 (640h29m4.410129028s) [noise=true , address=145.79.227.34:4924 , piece_id=QUPSGORFPMQVD2IRAK7COO2EE7XDV2KWEBTI2FQ2C25RTAAFNSRA, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.544558039 (640h29m4.415641084s) [noise=true , address=145.79.227.30:4784 , piece_id=HA6Z6M3HASDLIDP2PHS7MUDOHBF6T6ZNPTABXF5PYXR2HFYSBOPQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.544582956 (640h29m4.415665991s) [seg_pos={0 0}, upload=561203]: Segment reached optimal threshold of 14 pieces.
Jan 1 07:58:04.544811462 (640h29m4.415894527s) [noise=true , address=193.149.164.143:4694 , piece_id=EUO4UWMROIFFAJECCFRJBQF4XV25VDKGKX3UF2PTUMTE5JBGD67Q, seg_pos={0 0}, upload=561203]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx): protocol: expected piece hash; context canceled; EOF
Jan 1 07:58:04.544821240 (640h29m4.415904275s) [noise=true , address=193.149.164.143:4694 , piece_id=EUO4UWMROIFFAJECCFRJBQF4XV25VDKGKX3UF2PTUMTE5JBGD67Q, seg_pos={0 0}, upload=561203]: Long tail cancellation due to optimal threshold
Jan 1 07:58:04.545116732 (640h29m4.416199767s) [noise=true , address=64.49.13.169:4705 , piece_id=2P3JANSNK64726X7IYJOIQOKAQ4P4LJ5A2GVPX6UXOWUDIYRPZMA, seg_pos={0 0}, upload=561203]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:45312->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:45312->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45312->64.49.13.169:4705: use of closed network connection
Jan 1 07:58:04.545123666 (640h29m4.416206711s) [noise=true , address=64.49.13.169:4705 , piece_id=2P3JANSNK64726X7IYJOIQOKAQ4P4LJ5A2GVPX6UXOWUDIYRPZMA, seg_pos={0 0}, upload=561203]: Long tail cancellation due to optimal threshold
Jan 1 07:58:04.545303088 (640h29m4.416386123s) [noise=true , address=145.79.227.40:4066 , piece_id=ABZ7356MEBO4XPYAKFSS4EQJYJN4CWNJR3C6NXFWLAEO65KL3J5Q, seg_pos={0 0}, upload=561203]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF
Jan 1 07:58:04.545312055 (640h29m4.4163951s) [noise=true , address=145.79.227.40:4066 , piece_id=ABZ7356MEBO4XPYAKFSS4EQJYJN4CWNJR3C6NXFWLAEO65KL3J5Q, seg_pos={0 0}, upload=561203]: Long tail cancellation due to optimal threshold
Jan 1 07:58:04.545414279 (640h29m4.416497314s) [noise=true , address=64.49.13.137:4494 , piece_id=GCFOH6XRKDEGNG6TFPCMHG5RUCP2LUFS3XEOPNF42VYTGS63ZHQQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TX66fDVv3qu4yBGx9FNZTiArNeUnqxjMYwb3VGqYCQRe8RwoU): protocol: write tcp 188.166.225.170:35392->64.49.13.137:4494: use of closed network connection; write tcp 188.166.225.170:35392->64.49.13.137:4494: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35392->64.49.13.137:4494: use of closed network connection
Jan 1 07:58:04.545422815 (640h29m4.41650585s) [noise=true , address=64.49.13.137:4494 , piece_id=GCFOH6XRKDEGNG6TFPCMHG5RUCP2LUFS3XEOPNF42VYTGS63ZHQQ, seg_pos={0 0}, upload=561203]: Long tail cancellation due to optimal threshold
Jan 1 07:58:04.545439939 (640h29m4.416522974s) [seg_pos={0 0}, upload=561203]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 1 07:58:04.545580406 (640h29m4.416663441s) [upload=561203]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 1 07:58:04.545587800 (640h29m4.416670845s) [upload=561203]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 1 07:58:04.840726371 (640h29m4.711809416s) [upload=561203]: Done uploading segments...
Jan 1 07:58:04.840749646 (640h29m4.711832691s) [upload=561203]: Upload finished. err: <nil>
Duration: 21.202021137s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
12.490005154s trace:
https://jaeger.dev.storj.io/trace/44e9d74e3b154e8e
FAIL (2026-01-01 08:02:13 UTC)
orbiter warning: context deadline exceeded
Duration: 12.490101268s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [.................X.....X..XX.........................XX.....] duration:
15.470169137s trace:
https://jaeger.dev.storj.io/trace/10d96081b66cd020
FAIL (2026-01-01 08:01:59 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
73329s) [noise=true , address=145.79.227.40:4066 , piece_id=ABZ7356MEBO4XPYAKFSS4EQJYJN4CWNJR3C6NXFWLAEO65KL3J5Q, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.722330803 (640h29m1.593413848s) [noise=true , address=145.79.227.9:4391 , piece_id=XS6HYQTJO355YLXEIDBKHTJ44UAT5UBKHDUCFA744X6TBPGYN5YQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.722848709 (640h29m1.593931754s) [noise=true , address=77.90.31.12:4606 , piece_id=K4E5ECNMOMKEIFHZ3EZ2QSQHZ5IYP3CAD5DA6DBQLM4JYMOYU7MA, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.723452529 (640h29m1.594535574s) [noise=true , address=193.149.164.143:4694 , piece_id=EUO4UWMROIFFAJECCFRJBQF4XV25VDKGKX3UF2PTUMTE5JBGD67Q, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.723730728 (640h29m1.594813773s) [noise=true , address=45.91.116.5:4320 , piece_id=FOFLP7277VUUPTRZZHQLNVJ3YVYBELJTUYL7PTT7FJNN7J7QLS2A, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.724307808 (640h29m1.595390853s) [noise=true , address=64.49.13.169:4705 , piece_id=2P3JANSNK64726X7IYJOIQOKAQ4P4LJ5A2GVPX6UXOWUDIYRPZMA, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.724847144 (640h29m1.595930189s) [noise=true , address=64.49.13.137:4494 , piece_id=GCFOH6XRKDEGNG6TFPCMHG5RUCP2LUFS3XEOPNF42VYTGS63ZHQQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.724970960 (640h29m1.596054005s) [noise=true , address=77.90.31.15:4725 , piece_id=LLGZBPL7CHDI2TAY2SYMUMC264USSUQRVFR5P4HTLBWPBOTSUSOA, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.725289036 (640h29m1.596372071s) [noise=true , address=145.79.227.49:4371 , piece_id=RGZZMDFFVVSIQJ2ZEUSK275SBWMGV75WGNDLY6LJ2TNRPUSY7RMQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.725518012 (640h29m1.596601047s) [noise=true , address=77.90.31.13:4649 , piece_id=YRKZDAT5O2USQM4SO5GRZ45SX7MA3U52KN2CRGYPX5WJ4HIKOJBA, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.725902264 (640h29m1.596985309s) [noise=true , address=45.91.116.12:4238 , piece_id=R5LPEYTYC3XB76SSMDVHPZORAWB2P3S4TETMYV23NJI4UXYUYCEQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.726176987 (640h29m1.597260032s) [noise=true , address=145.79.227.35:4497 , piece_id=YDNRFAJVXHRJD5QHICHR6CGF6K5ARZTLHOUSRLV6JAYU3ZSWEY3A, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.726843697 (640h29m1.597926742s) [noise=true , address=77.90.31.19:4846 , piece_id=2XJSN2WTR2BAVKDJZXEMY6SRX7KLOU4SVGYH7TNJIJTPW723DEHQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.727470170 (640h29m1.598553225s) [noise=true , address=145.79.227.30:4784 , piece_id=HA6Z6M3HASDLIDP2PHS7MUDOHBF6T6ZNPTABXF5PYXR2HFYSBOPQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.728088075 (640h29m1.59917112s) [noise=true , address=145.79.227.39:4032 , piece_id=Z7BQPDXOVQ6AEKACBB2DJFI652AT6UIHXUB2HPUJFMK53EF73AWQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:03.152355605 (640h29m3.02343864s) [upload=561203]: Next returned nil segment
Jan 1 07:58:03.152371765 (640h29m3.023454811s) [upload=561203]: Waiting for error group managing segments...
Jan 1 07:58:03.563991563 (640h29m3.435074608s) [noise=true , address=145.79.227.9:4391 , piece_id=XS6HYQTJO355YLXEIDBKHTJ44UAT5UBKHDUCFA744X6TBPGYN5YQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:03.652285026 (640h29m3.523368071s) [noise=true , address=145.79.227.49:4371 , piece_id=RGZZMDFFVVSIQJ2ZEUSK275SBWMGV75WGNDLY6LJ2TNRPUSY7RMQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:03.671359306 (640h29m3.542442351s) [noise=true , address=145.79.227.39:4032 , piece_id=Z7BQPDXOVQ6AEKACBB2DJFI652AT6UIHXUB2HPUJFMK53EF73AWQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:03.774991565 (640h29m3.64607462s) [noise=true , address=77.90.31.19:4846 , piece_id=2XJSN2WTR2BAVKDJZXEMY6SRX7KLOU4SVGYH7TNJIJTPW723DEHQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:03.870781221 (640h29m3.741864266s) [noise=true , address=77.90.31.12:4606 , piece_id=K4E5ECNMOMKEIFHZ3EZ2QSQHZ5IYP3CAD5DA6DBQLM4JYMOYU7MA, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:03.878179488 (640h29m3.749262533s) [noise=true , address=77.90.31.13:4649 , piece_id=YRKZDAT5O2USQM4SO5GRZ45SX7MA3U52KN2CRGYPX5WJ4HIKOJBA, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:03.977331467 (640h29m3.848414512s) [noise=true , address=77.90.31.15:4725 , piece_id=LLGZBPL7CHDI2TAY2SYMUMC264USSUQRVFR5P4HTLBWPBOTSUSOA, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.224768738 (640h29m4.095851783s) [noise=true , address=45.91.116.12:4238 , piece_id=R5LPEYTYC3XB76SSMDVHPZORAWB2P3S4TETMYV23NJI4UXYUYCEQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.457462948 (640h29m4.328545994s) [noise=true , address=45.91.116.5:4320 , piece_id=FOFLP7277VUUPTRZZHQLNVJ3YVYBELJTUYL7PTT7FJNN7J7QLS2A, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.488291199 (640h29m4.359374234s) [noise=true , address=145.79.227.35:4497 , piece_id=YDNRFAJVXHRJD5QHICHR6CGF6K5ARZTLHOUSRLV6JAYU3ZSWEY3A, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.522541837 (640h29m4.393624882s) [noise=true , address=145.79.227.43:4171 , piece_id=KYGE2RFI6O6DUFWVGJV2IT5VE3U43WJ4ODNQMTZFQH2DAADFC6BQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.525844266 (640h29m4.396927321s) [noise=true , address=145.79.227.33:4886 , piece_id=CYGCX4TM2ZFPCJYVDF52PDNR2OCNZ2NUSUUOUMUHYGCADWWFDHVQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.539045983 (640h29m4.410129028s) [noise=true , address=145.79.227.34:4924 , piece_id=QUPSGORFPMQVD2IRAK7COO2EE7XDV2KWEBTI2FQ2C25RTAAFNSRA, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.544558039 (640h29m4.415641084s) [noise=true , address=145.79.227.30:4784 , piece_id=HA6Z6M3HASDLIDP2PHS7MUDOHBF6T6ZNPTABXF5PYXR2HFYSBOPQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.544582956 (640h29m4.415665991s) [seg_pos={0 0}, upload=561203]: Segment reached optimal threshold of 14 pieces.
Jan 1 07:58:04.544811462 (640h29m4.415894527s) [noise=true , address=193.149.164.143:4694 , piece_id=EUO4UWMROIFFAJECCFRJBQF4XV25VDKGKX3UF2PTUMTE5JBGD67Q, seg_pos={0 0}, upload=561203]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx): protocol: expected piece hash; context canceled; EOF
Jan 1 07:58:04.544821240 (640h29m4.415904275s) [noise=true , address=193.149.164.143:4694 , piece_id=EUO4UWMROIFFAJECCFRJBQF4XV25VDKGKX3UF2PTUMTE5JBGD67Q, seg_pos={0 0}, upload=561203]: Long tail cancellation due to optimal threshold
Jan 1 07:58:04.545116732 (640h29m4.416199767s) [noise=true , address=64.49.13.169:4705 , piece_id=2P3JANSNK64726X7IYJOIQOKAQ4P4LJ5A2GVPX6UXOWUDIYRPZMA, seg_pos={0 0}, upload=561203]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:45312->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:45312->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45312->64.49.13.169:4705: use of closed network connection
Jan 1 07:58:04.545123666 (640h29m4.416206711s) [noise=true , address=64.49.13.169:4705 , piece_id=2P3JANSNK64726X7IYJOIQOKAQ4P4LJ5A2GVPX6UXOWUDIYRPZMA, seg_pos={0 0}, upload=561203]: Long tail cancellation due to optimal threshold
Jan 1 07:58:04.545303088 (640h29m4.416386123s) [noise=true , address=145.79.227.40:4066 , piece_id=ABZ7356MEBO4XPYAKFSS4EQJYJN4CWNJR3C6NXFWLAEO65KL3J5Q, seg_pos={0 0}, upload=561203]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF
Jan 1 07:58:04.545312055 (640h29m4.4163951s) [noise=true , address=145.79.227.40:4066 , piece_id=ABZ7356MEBO4XPYAKFSS4EQJYJN4CWNJR3C6NXFWLAEO65KL3J5Q, seg_pos={0 0}, upload=561203]: Long tail cancellation due to optimal threshold
Jan 1 07:58:04.545414279 (640h29m4.416497314s) [noise=true , address=64.49.13.137:4494 , piece_id=GCFOH6XRKDEGNG6TFPCMHG5RUCP2LUFS3XEOPNF42VYTGS63ZHQQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TX66fDVv3qu4yBGx9FNZTiArNeUnqxjMYwb3VGqYCQRe8RwoU): protocol: write tcp 188.166.225.170:35392->64.49.13.137:4494: use of closed network connection; write tcp 188.166.225.170:35392->64.49.13.137:4494: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35392->64.49.13.137:4494: use of closed network connection
Jan 1 07:58:04.545422815 (640h29m4.41650585s) [noise=true , address=64.49.13.137:4494 , piece_id=GCFOH6XRKDEGNG6TFPCMHG5RUCP2LUFS3XEOPNF42VYTGS63ZHQQ, seg_pos={0 0}, upload=561203]: Long tail cancellation due to optimal threshold
Jan 1 07:58:04.545439939 (640h29m4.416522974s) [seg_pos={0 0}, upload=561203]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 1 07:58:04.545580406 (640h29m4.416663441s) [upload=561203]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 1 07:58:04.545587800 (640h29m4.416670845s) [upload=561203]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 1 07:58:04.840726371 (640h29m4.711809416s) [upload=561203]: Done uploading segments...
Jan 1 07:58:04.840749646 (640h29m4.711832691s) [upload=561203]: Upload finished. err: <nil>
Duration: 15.470195346s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
21.424301298s trace:
https://jaeger.dev.storj.io/trace/44e9d74e3b154e8e
FAIL (2026-01-01 08:01:43 UTC)
orbiter warning: context deadline exceeded
Duration: 21.424326185s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [..................X.....X..XX.........................XX....] duration:
15.450565424s trace:
https://jaeger.dev.storj.io/trace/10d96081b66cd020
FAIL (2026-01-01 08:01:41 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
73329s) [noise=true , address=145.79.227.40:4066 , piece_id=ABZ7356MEBO4XPYAKFSS4EQJYJN4CWNJR3C6NXFWLAEO65KL3J5Q, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.722330803 (640h29m1.593413848s) [noise=true , address=145.79.227.9:4391 , piece_id=XS6HYQTJO355YLXEIDBKHTJ44UAT5UBKHDUCFA744X6TBPGYN5YQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.722848709 (640h29m1.593931754s) [noise=true , address=77.90.31.12:4606 , piece_id=K4E5ECNMOMKEIFHZ3EZ2QSQHZ5IYP3CAD5DA6DBQLM4JYMOYU7MA, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.723452529 (640h29m1.594535574s) [noise=true , address=193.149.164.143:4694 , piece_id=EUO4UWMROIFFAJECCFRJBQF4XV25VDKGKX3UF2PTUMTE5JBGD67Q, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.723730728 (640h29m1.594813773s) [noise=true , address=45.91.116.5:4320 , piece_id=FOFLP7277VUUPTRZZHQLNVJ3YVYBELJTUYL7PTT7FJNN7J7QLS2A, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.724307808 (640h29m1.595390853s) [noise=true , address=64.49.13.169:4705 , piece_id=2P3JANSNK64726X7IYJOIQOKAQ4P4LJ5A2GVPX6UXOWUDIYRPZMA, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.724847144 (640h29m1.595930189s) [noise=true , address=64.49.13.137:4494 , piece_id=GCFOH6XRKDEGNG6TFPCMHG5RUCP2LUFS3XEOPNF42VYTGS63ZHQQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.724970960 (640h29m1.596054005s) [noise=true , address=77.90.31.15:4725 , piece_id=LLGZBPL7CHDI2TAY2SYMUMC264USSUQRVFR5P4HTLBWPBOTSUSOA, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.725289036 (640h29m1.596372071s) [noise=true , address=145.79.227.49:4371 , piece_id=RGZZMDFFVVSIQJ2ZEUSK275SBWMGV75WGNDLY6LJ2TNRPUSY7RMQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.725518012 (640h29m1.596601047s) [noise=true , address=77.90.31.13:4649 , piece_id=YRKZDAT5O2USQM4SO5GRZ45SX7MA3U52KN2CRGYPX5WJ4HIKOJBA, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.725902264 (640h29m1.596985309s) [noise=true , address=45.91.116.12:4238 , piece_id=R5LPEYTYC3XB76SSMDVHPZORAWB2P3S4TETMYV23NJI4UXYUYCEQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.726176987 (640h29m1.597260032s) [noise=true , address=145.79.227.35:4497 , piece_id=YDNRFAJVXHRJD5QHICHR6CGF6K5ARZTLHOUSRLV6JAYU3ZSWEY3A, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.726843697 (640h29m1.597926742s) [noise=true , address=77.90.31.19:4846 , piece_id=2XJSN2WTR2BAVKDJZXEMY6SRX7KLOU4SVGYH7TNJIJTPW723DEHQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.727470170 (640h29m1.598553225s) [noise=true , address=145.79.227.30:4784 , piece_id=HA6Z6M3HASDLIDP2PHS7MUDOHBF6T6ZNPTABXF5PYXR2HFYSBOPQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.728088075 (640h29m1.59917112s) [noise=true , address=145.79.227.39:4032 , piece_id=Z7BQPDXOVQ6AEKACBB2DJFI652AT6UIHXUB2HPUJFMK53EF73AWQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:03.152355605 (640h29m3.02343864s) [upload=561203]: Next returned nil segment
Jan 1 07:58:03.152371765 (640h29m3.023454811s) [upload=561203]: Waiting for error group managing segments...
Jan 1 07:58:03.563991563 (640h29m3.435074608s) [noise=true , address=145.79.227.9:4391 , piece_id=XS6HYQTJO355YLXEIDBKHTJ44UAT5UBKHDUCFA744X6TBPGYN5YQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:03.652285026 (640h29m3.523368071s) [noise=true , address=145.79.227.49:4371 , piece_id=RGZZMDFFVVSIQJ2ZEUSK275SBWMGV75WGNDLY6LJ2TNRPUSY7RMQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:03.671359306 (640h29m3.542442351s) [noise=true , address=145.79.227.39:4032 , piece_id=Z7BQPDXOVQ6AEKACBB2DJFI652AT6UIHXUB2HPUJFMK53EF73AWQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:03.774991565 (640h29m3.64607462s) [noise=true , address=77.90.31.19:4846 , piece_id=2XJSN2WTR2BAVKDJZXEMY6SRX7KLOU4SVGYH7TNJIJTPW723DEHQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:03.870781221 (640h29m3.741864266s) [noise=true , address=77.90.31.12:4606 , piece_id=K4E5ECNMOMKEIFHZ3EZ2QSQHZ5IYP3CAD5DA6DBQLM4JYMOYU7MA, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:03.878179488 (640h29m3.749262533s) [noise=true , address=77.90.31.13:4649 , piece_id=YRKZDAT5O2USQM4SO5GRZ45SX7MA3U52KN2CRGYPX5WJ4HIKOJBA, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:03.977331467 (640h29m3.848414512s) [noise=true , address=77.90.31.15:4725 , piece_id=LLGZBPL7CHDI2TAY2SYMUMC264USSUQRVFR5P4HTLBWPBOTSUSOA, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.224768738 (640h29m4.095851783s) [noise=true , address=45.91.116.12:4238 , piece_id=R5LPEYTYC3XB76SSMDVHPZORAWB2P3S4TETMYV23NJI4UXYUYCEQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.457462948 (640h29m4.328545994s) [noise=true , address=45.91.116.5:4320 , piece_id=FOFLP7277VUUPTRZZHQLNVJ3YVYBELJTUYL7PTT7FJNN7J7QLS2A, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.488291199 (640h29m4.359374234s) [noise=true , address=145.79.227.35:4497 , piece_id=YDNRFAJVXHRJD5QHICHR6CGF6K5ARZTLHOUSRLV6JAYU3ZSWEY3A, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.522541837 (640h29m4.393624882s) [noise=true , address=145.79.227.43:4171 , piece_id=KYGE2RFI6O6DUFWVGJV2IT5VE3U43WJ4ODNQMTZFQH2DAADFC6BQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.525844266 (640h29m4.396927321s) [noise=true , address=145.79.227.33:4886 , piece_id=CYGCX4TM2ZFPCJYVDF52PDNR2OCNZ2NUSUUOUMUHYGCADWWFDHVQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.539045983 (640h29m4.410129028s) [noise=true , address=145.79.227.34:4924 , piece_id=QUPSGORFPMQVD2IRAK7COO2EE7XDV2KWEBTI2FQ2C25RTAAFNSRA, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.544558039 (640h29m4.415641084s) [noise=true , address=145.79.227.30:4784 , piece_id=HA6Z6M3HASDLIDP2PHS7MUDOHBF6T6ZNPTABXF5PYXR2HFYSBOPQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.544582956 (640h29m4.415665991s) [seg_pos={0 0}, upload=561203]: Segment reached optimal threshold of 14 pieces.
Jan 1 07:58:04.544811462 (640h29m4.415894527s) [noise=true , address=193.149.164.143:4694 , piece_id=EUO4UWMROIFFAJECCFRJBQF4XV25VDKGKX3UF2PTUMTE5JBGD67Q, seg_pos={0 0}, upload=561203]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx): protocol: expected piece hash; context canceled; EOF
Jan 1 07:58:04.544821240 (640h29m4.415904275s) [noise=true , address=193.149.164.143:4694 , piece_id=EUO4UWMROIFFAJECCFRJBQF4XV25VDKGKX3UF2PTUMTE5JBGD67Q, seg_pos={0 0}, upload=561203]: Long tail cancellation due to optimal threshold
Jan 1 07:58:04.545116732 (640h29m4.416199767s) [noise=true , address=64.49.13.169:4705 , piece_id=2P3JANSNK64726X7IYJOIQOKAQ4P4LJ5A2GVPX6UXOWUDIYRPZMA, seg_pos={0 0}, upload=561203]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:45312->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:45312->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45312->64.49.13.169:4705: use of closed network connection
Jan 1 07:58:04.545123666 (640h29m4.416206711s) [noise=true , address=64.49.13.169:4705 , piece_id=2P3JANSNK64726X7IYJOIQOKAQ4P4LJ5A2GVPX6UXOWUDIYRPZMA, seg_pos={0 0}, upload=561203]: Long tail cancellation due to optimal threshold
Jan 1 07:58:04.545303088 (640h29m4.416386123s) [noise=true , address=145.79.227.40:4066 , piece_id=ABZ7356MEBO4XPYAKFSS4EQJYJN4CWNJR3C6NXFWLAEO65KL3J5Q, seg_pos={0 0}, upload=561203]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF
Jan 1 07:58:04.545312055 (640h29m4.4163951s) [noise=true , address=145.79.227.40:4066 , piece_id=ABZ7356MEBO4XPYAKFSS4EQJYJN4CWNJR3C6NXFWLAEO65KL3J5Q, seg_pos={0 0}, upload=561203]: Long tail cancellation due to optimal threshold
Jan 1 07:58:04.545414279 (640h29m4.416497314s) [noise=true , address=64.49.13.137:4494 , piece_id=GCFOH6XRKDEGNG6TFPCMHG5RUCP2LUFS3XEOPNF42VYTGS63ZHQQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TX66fDVv3qu4yBGx9FNZTiArNeUnqxjMYwb3VGqYCQRe8RwoU): protocol: write tcp 188.166.225.170:35392->64.49.13.137:4494: use of closed network connection; write tcp 188.166.225.170:35392->64.49.13.137:4494: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35392->64.49.13.137:4494: use of closed network connection
Jan 1 07:58:04.545422815 (640h29m4.41650585s) [noise=true , address=64.49.13.137:4494 , piece_id=GCFOH6XRKDEGNG6TFPCMHG5RUCP2LUFS3XEOPNF42VYTGS63ZHQQ, seg_pos={0 0}, upload=561203]: Long tail cancellation due to optimal threshold
Jan 1 07:58:04.545439939 (640h29m4.416522974s) [seg_pos={0 0}, upload=561203]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 1 07:58:04.545580406 (640h29m4.416663441s) [upload=561203]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 1 07:58:04.545587800 (640h29m4.416670845s) [upload=561203]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 1 07:58:04.840726371 (640h29m4.711809416s) [upload=561203]: Done uploading segments...
Jan 1 07:58:04.840749646 (640h29m4.711832691s) [upload=561203]: Upload finished. err: <nil>
Duration: 15.450579309s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
37 out of 60 is failed (threshold=6) [..XX.XX.X.XXXXX.XX..XXXXX.XXXX..XX.XXX.XX...X.X..X...XXXXXX.] duration:
3.726225183s trace:
https://jaeger.dev.storj.io/trace/5195a3bdc9c8f86b
FAIL (2026-01-01 08:01:41 UTC)
orbiter warning: context deadline exceeded
Duration: 3.726254308s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [...................X.....X..XX.........................XX...] duration:
50.422050529s trace:
https://jaeger.dev.storj.io/trace/10d96081b66cd020
FAIL (2026-01-01 08:01:25 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
73329s) [noise=true , address=145.79.227.40:4066 , piece_id=ABZ7356MEBO4XPYAKFSS4EQJYJN4CWNJR3C6NXFWLAEO65KL3J5Q, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.722330803 (640h29m1.593413848s) [noise=true , address=145.79.227.9:4391 , piece_id=XS6HYQTJO355YLXEIDBKHTJ44UAT5UBKHDUCFA744X6TBPGYN5YQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.722848709 (640h29m1.593931754s) [noise=true , address=77.90.31.12:4606 , piece_id=K4E5ECNMOMKEIFHZ3EZ2QSQHZ5IYP3CAD5DA6DBQLM4JYMOYU7MA, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.723452529 (640h29m1.594535574s) [noise=true , address=193.149.164.143:4694 , piece_id=EUO4UWMROIFFAJECCFRJBQF4XV25VDKGKX3UF2PTUMTE5JBGD67Q, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.723730728 (640h29m1.594813773s) [noise=true , address=45.91.116.5:4320 , piece_id=FOFLP7277VUUPTRZZHQLNVJ3YVYBELJTUYL7PTT7FJNN7J7QLS2A, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.724307808 (640h29m1.595390853s) [noise=true , address=64.49.13.169:4705 , piece_id=2P3JANSNK64726X7IYJOIQOKAQ4P4LJ5A2GVPX6UXOWUDIYRPZMA, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.724847144 (640h29m1.595930189s) [noise=true , address=64.49.13.137:4494 , piece_id=GCFOH6XRKDEGNG6TFPCMHG5RUCP2LUFS3XEOPNF42VYTGS63ZHQQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.724970960 (640h29m1.596054005s) [noise=true , address=77.90.31.15:4725 , piece_id=LLGZBPL7CHDI2TAY2SYMUMC264USSUQRVFR5P4HTLBWPBOTSUSOA, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.725289036 (640h29m1.596372071s) [noise=true , address=145.79.227.49:4371 , piece_id=RGZZMDFFVVSIQJ2ZEUSK275SBWMGV75WGNDLY6LJ2TNRPUSY7RMQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.725518012 (640h29m1.596601047s) [noise=true , address=77.90.31.13:4649 , piece_id=YRKZDAT5O2USQM4SO5GRZ45SX7MA3U52KN2CRGYPX5WJ4HIKOJBA, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.725902264 (640h29m1.596985309s) [noise=true , address=45.91.116.12:4238 , piece_id=R5LPEYTYC3XB76SSMDVHPZORAWB2P3S4TETMYV23NJI4UXYUYCEQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.726176987 (640h29m1.597260032s) [noise=true , address=145.79.227.35:4497 , piece_id=YDNRFAJVXHRJD5QHICHR6CGF6K5ARZTLHOUSRLV6JAYU3ZSWEY3A, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.726843697 (640h29m1.597926742s) [noise=true , address=77.90.31.19:4846 , piece_id=2XJSN2WTR2BAVKDJZXEMY6SRX7KLOU4SVGYH7TNJIJTPW723DEHQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.727470170 (640h29m1.598553225s) [noise=true , address=145.79.227.30:4784 , piece_id=HA6Z6M3HASDLIDP2PHS7MUDOHBF6T6ZNPTABXF5PYXR2HFYSBOPQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.728088075 (640h29m1.59917112s) [noise=true , address=145.79.227.39:4032 , piece_id=Z7BQPDXOVQ6AEKACBB2DJFI652AT6UIHXUB2HPUJFMK53EF73AWQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:03.152355605 (640h29m3.02343864s) [upload=561203]: Next returned nil segment
Jan 1 07:58:03.152371765 (640h29m3.023454811s) [upload=561203]: Waiting for error group managing segments...
Jan 1 07:58:03.563991563 (640h29m3.435074608s) [noise=true , address=145.79.227.9:4391 , piece_id=XS6HYQTJO355YLXEIDBKHTJ44UAT5UBKHDUCFA744X6TBPGYN5YQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:03.652285026 (640h29m3.523368071s) [noise=true , address=145.79.227.49:4371 , piece_id=RGZZMDFFVVSIQJ2ZEUSK275SBWMGV75WGNDLY6LJ2TNRPUSY7RMQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:03.671359306 (640h29m3.542442351s) [noise=true , address=145.79.227.39:4032 , piece_id=Z7BQPDXOVQ6AEKACBB2DJFI652AT6UIHXUB2HPUJFMK53EF73AWQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:03.774991565 (640h29m3.64607462s) [noise=true , address=77.90.31.19:4846 , piece_id=2XJSN2WTR2BAVKDJZXEMY6SRX7KLOU4SVGYH7TNJIJTPW723DEHQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:03.870781221 (640h29m3.741864266s) [noise=true , address=77.90.31.12:4606 , piece_id=K4E5ECNMOMKEIFHZ3EZ2QSQHZ5IYP3CAD5DA6DBQLM4JYMOYU7MA, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:03.878179488 (640h29m3.749262533s) [noise=true , address=77.90.31.13:4649 , piece_id=YRKZDAT5O2USQM4SO5GRZ45SX7MA3U52KN2CRGYPX5WJ4HIKOJBA, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:03.977331467 (640h29m3.848414512s) [noise=true , address=77.90.31.15:4725 , piece_id=LLGZBPL7CHDI2TAY2SYMUMC264USSUQRVFR5P4HTLBWPBOTSUSOA, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.224768738 (640h29m4.095851783s) [noise=true , address=45.91.116.12:4238 , piece_id=R5LPEYTYC3XB76SSMDVHPZORAWB2P3S4TETMYV23NJI4UXYUYCEQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.457462948 (640h29m4.328545994s) [noise=true , address=45.91.116.5:4320 , piece_id=FOFLP7277VUUPTRZZHQLNVJ3YVYBELJTUYL7PTT7FJNN7J7QLS2A, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.488291199 (640h29m4.359374234s) [noise=true , address=145.79.227.35:4497 , piece_id=YDNRFAJVXHRJD5QHICHR6CGF6K5ARZTLHOUSRLV6JAYU3ZSWEY3A, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.522541837 (640h29m4.393624882s) [noise=true , address=145.79.227.43:4171 , piece_id=KYGE2RFI6O6DUFWVGJV2IT5VE3U43WJ4ODNQMTZFQH2DAADFC6BQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.525844266 (640h29m4.396927321s) [noise=true , address=145.79.227.33:4886 , piece_id=CYGCX4TM2ZFPCJYVDF52PDNR2OCNZ2NUSUUOUMUHYGCADWWFDHVQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.539045983 (640h29m4.410129028s) [noise=true , address=145.79.227.34:4924 , piece_id=QUPSGORFPMQVD2IRAK7COO2EE7XDV2KWEBTI2FQ2C25RTAAFNSRA, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.544558039 (640h29m4.415641084s) [noise=true , address=145.79.227.30:4784 , piece_id=HA6Z6M3HASDLIDP2PHS7MUDOHBF6T6ZNPTABXF5PYXR2HFYSBOPQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.544582956 (640h29m4.415665991s) [seg_pos={0 0}, upload=561203]: Segment reached optimal threshold of 14 pieces.
Jan 1 07:58:04.544811462 (640h29m4.415894527s) [noise=true , address=193.149.164.143:4694 , piece_id=EUO4UWMROIFFAJECCFRJBQF4XV25VDKGKX3UF2PTUMTE5JBGD67Q, seg_pos={0 0}, upload=561203]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx): protocol: expected piece hash; context canceled; EOF
Jan 1 07:58:04.544821240 (640h29m4.415904275s) [noise=true , address=193.149.164.143:4694 , piece_id=EUO4UWMROIFFAJECCFRJBQF4XV25VDKGKX3UF2PTUMTE5JBGD67Q, seg_pos={0 0}, upload=561203]: Long tail cancellation due to optimal threshold
Jan 1 07:58:04.545116732 (640h29m4.416199767s) [noise=true , address=64.49.13.169:4705 , piece_id=2P3JANSNK64726X7IYJOIQOKAQ4P4LJ5A2GVPX6UXOWUDIYRPZMA, seg_pos={0 0}, upload=561203]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:45312->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:45312->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45312->64.49.13.169:4705: use of closed network connection
Jan 1 07:58:04.545123666 (640h29m4.416206711s) [noise=true , address=64.49.13.169:4705 , piece_id=2P3JANSNK64726X7IYJOIQOKAQ4P4LJ5A2GVPX6UXOWUDIYRPZMA, seg_pos={0 0}, upload=561203]: Long tail cancellation due to optimal threshold
Jan 1 07:58:04.545303088 (640h29m4.416386123s) [noise=true , address=145.79.227.40:4066 , piece_id=ABZ7356MEBO4XPYAKFSS4EQJYJN4CWNJR3C6NXFWLAEO65KL3J5Q, seg_pos={0 0}, upload=561203]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF
Jan 1 07:58:04.545312055 (640h29m4.4163951s) [noise=true , address=145.79.227.40:4066 , piece_id=ABZ7356MEBO4XPYAKFSS4EQJYJN4CWNJR3C6NXFWLAEO65KL3J5Q, seg_pos={0 0}, upload=561203]: Long tail cancellation due to optimal threshold
Jan 1 07:58:04.545414279 (640h29m4.416497314s) [noise=true , address=64.49.13.137:4494 , piece_id=GCFOH6XRKDEGNG6TFPCMHG5RUCP2LUFS3XEOPNF42VYTGS63ZHQQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TX66fDVv3qu4yBGx9FNZTiArNeUnqxjMYwb3VGqYCQRe8RwoU): protocol: write tcp 188.166.225.170:35392->64.49.13.137:4494: use of closed network connection; write tcp 188.166.225.170:35392->64.49.13.137:4494: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35392->64.49.13.137:4494: use of closed network connection
Jan 1 07:58:04.545422815 (640h29m4.41650585s) [noise=true , address=64.49.13.137:4494 , piece_id=GCFOH6XRKDEGNG6TFPCMHG5RUCP2LUFS3XEOPNF42VYTGS63ZHQQ, seg_pos={0 0}, upload=561203]: Long tail cancellation due to optimal threshold
Jan 1 07:58:04.545439939 (640h29m4.416522974s) [seg_pos={0 0}, upload=561203]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 1 07:58:04.545580406 (640h29m4.416663441s) [upload=561203]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 1 07:58:04.545587800 (640h29m4.416670845s) [upload=561203]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 1 07:58:04.840726371 (640h29m4.711809416s) [upload=561203]: Done uploading segments...
Jan 1 07:58:04.840749646 (640h29m4.711832691s) [upload=561203]: Upload finished. err: <nil>
Duration: 50.422068773s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
37 out of 60 is failed (threshold=6) [...XX.XX.X.XXXXX.XX..XXXXX.XXXX..XX.XXX.XX...X.X..X...XXXXXX] duration:
5.000979873s trace:
https://jaeger.dev.storj.io/trace/5195a3bdc9c8f86b
FAIL (2026-01-01 08:01:19 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001010751s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [...X.X.XX.XX..X...X.XX.X...X.........X.X........XX.X...X..X.] duration:
6.68428084s trace:
https://jaeger.dev.storj.io/trace/14a6f6c583f5aa92
FAIL (2026-01-01 08:01:13 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: BPDHJMIUVCABD22C32WNGZZLNGGMWHI46PMQ7MBXBEDYPMTYPRZQ) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: ZN4WX6RP777Z6PHCE6UZOKKCK72YOXTCIHBSGNCWGMVSCC3ZW4KA) context deadline exceeded
======LOGS======
Jan 1 08:00:36.336451400 (640h31m36.207534445s) [upload=561245]: Starting upload
Jan 1 08:00:36.336534237 (640h31m36.207617283s) [upload=561245]: Done starting upload
Jan 1 08:00:36.336736032 (640h31m36.207819077s) [upload=561245]: Uploading segments...
Jan 1 08:00:36.336819661 (640h31m36.207902706s) [upload=561245]: Got next segment. Inline: false
Jan 1 08:00:36.336868724 (640h31m36.207951759s) [upload=561245]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 1 08:00:36.336875117 (640h31m36.207958162s) [upload=561245]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 1 08:00:36.580601654 (640h31m36.451684689s) [seg_pos={0 0}, upload=561245]: Begin upload segment...
Jan 1 08:00:36.580652981 (640h31m36.451736026s) [seg_pos={0 0}, upload=561245]: Done begin upload segment.
Jan 1 08:00:36.580707806 (640h31m36.451790851s) [noise=true , address=77.90.31.19:4841 , piece_id=W4EMUECOZ6NXFBZROVL6W6SI2AADAT6QZMKVZWFUOOJ6RPUHOGQQ, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.580843394 (640h31m36.451926429s) [noise=true , address=145.79.227.21:4475 , piece_id=H6RSWRNNW7U5XNKC27OEKZ5ILDOX73OCOZ6N2L35RKZ6TSZPHNFQ, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.581549899 (640h31m36.452632944s) [noise=true , address=145.79.227.29:4942 , piece_id=WDSA5DJWK3OVIAOAUCR4I2VTSG4HD4UT7EMV7SECRQHWUPZV2Y4A, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.582165562 (640h31m36.453248597s) [noise=true , address=145.79.227.48:4302 , piece_id=NJ3UEHC2WBCH7KJO3WZVHXKP3FDLHXKQULVEL7YMNMBU4OIKZUOA, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.582576053 (640h31m36.453659088s) [noise=true , address=145.79.227.30:4762 , piece_id=WZI2E3HGEQXIMFQBWBZ7PK2W5NDZMNBCKRF3EDB4MSOCA3DGD7JQ, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.583033354 (640h31m36.454116399s) [noise=true , address=145.79.227.18:4007 , piece_id=G4NXSCJCZ6LN2RLQSFGQX4XKLNPZFFZUOFROM3SFMOGFUYLWRHNQ, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.583471328 (640h31m36.454554373s) [noise=true , address=45.91.116.5:4297 , piece_id=XEGWKAMHMY3CRSMXGNAL4CDFDEFIVEDQINC7TXKILYIV73QV4UZQ, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.584935826 (640h31m36.456018871s) [noise=true , address=77.90.31.14:4661 , piece_id=BPDHJMIUVCABD22C32WNGZZLNGGMWHI46PMQ7MBXBEDYPMTYPRZQ, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.585046487 (640h31m36.456129522s) [noise=true , address=145.79.227.14:4840 , piece_id=54YIHG6H34PFOQO4OU3MV74CZUCFY5N2OCATSIOCMD453XOUANWQ, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.586542315 (640h31m36.45762535s) [noise=true , address=145.79.227.45:4230 , piece_id=6S3NFOZFIRQUE3T3PEZLZLMNVIOK33JUSYVH5HVWEJ3RMSDKAFXQ, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.586809845 (640h31m36.45789288s) [noise=true , address=145.79.227.33:4870 , piece_id=OYMN4GE2TWKCSLFDJ66GJ6UB2ZXDHK7D4CC5A7AYH4NGJCAT66JA, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.588268963 (640h31m36.459351998s) [noise=true , address=145.79.227.24:4581 , piece_id=DRKKS2N35E7ORYSCARSNHBNKN3DTIGV5STIISCK7THZW76CDVO3A, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.590930822 (640h31m36.462013867s) [noise=true , address=145.79.227.38:4977 , piece_id=J3YUHNPAJEX5D76LVFHZBAP73HSG5IN7ZI5VD7IX4ZVMVGHSAHAA, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.594801081 (640h31m36.465884116s) [noise=true , address=145.79.227.52:4447 , piece_id=R7LXPRI2SWX2PAAALMPFOHP3JNJAFJ5QLJINAFKVK7H6PEE2TTGA, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.598974989 (640h31m36.470058024s) [noise=true , address=145.79.227.9:4367 , piece_id=JGIZMNV72PTWNVAKBO2CQMG6VGXJA6YUJABLSTORA3ZMDO6HYHRQ, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.600900144 (640h31m36.47198319s) [noise=true , address=45.91.116.8:4048 , piece_id=ZN4WX6RP777Z6PHCE6UZOKKCK72YOXTCIHBSGNCWGMVSCC3ZW4KA, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:38.299065741 (640h31m38.170148776s) [upload=561245]: Next returned nil segment
Jan 1 08:00:38.299077214 (640h31m38.170160259s) [upload=561245]: Waiting for error group managing segments...
Jan 1 08:00:38.586542564 (640h31m38.45762562s) [noise=true , address=145.79.227.21:4475 , piece_id=H6RSWRNNW7U5XNKC27OEKZ5ILDOX73OCOZ6N2L35RKZ6TSZPHNFQ, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:38.587267896 (640h31m38.458350941s) [noise=true , address=145.79.227.52:4447 , piece_id=R7LXPRI2SWX2PAAALMPFOHP3JNJAFJ5QLJINAFKVK7H6PEE2TTGA, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:38.587822492 (640h31m38.458905537s) [noise=true , address=145.79.227.18:4007 , piece_id=G4NXSCJCZ6LN2RLQSFGQX4XKLNPZFFZUOFROM3SFMOGFUYLWRHNQ, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:38.600692919 (640h31m38.471775964s) [noise=true , address=145.79.227.45:4230 , piece_id=6S3NFOZFIRQUE3T3PEZLZLMNVIOK33JUSYVH5HVWEJ3RMSDKAFXQ, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:38.601771583 (640h31m38.472854628s) [noise=true , address=145.79.227.48:4302 , piece_id=NJ3UEHC2WBCH7KJO3WZVHXKP3FDLHXKQULVEL7YMNMBU4OIKZUOA, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:38.606091358 (640h31m38.477174403s) [noise=true , address=145.79.227.38:4977 , piece_id=J3YUHNPAJEX5D76LVFHZBAP73HSG5IN7ZI5VD7IX4ZVMVGHSAHAA, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:38.611687373 (640h31m38.482770418s) [noise=true , address=145.79.227.14:4840 , piece_id=54YIHG6H34PFOQO4OU3MV74CZUCFY5N2OCATSIOCMD453XOUANWQ, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:38.614887266 (640h31m38.485970312s) [noise=true , address=145.79.227.24:4581 , piece_id=DRKKS2N35E7ORYSCARSNHBNKN3DTIGV5STIISCK7THZW76CDVO3A, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:38.626493386 (640h31m38.497576431s) [noise=true , address=145.79.227.9:4367 , piece_id=JGIZMNV72PTWNVAKBO2CQMG6VGXJA6YUJABLSTORA3ZMDO6HYHRQ, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:38.910948826 (640h31m38.782031871s) [noise=true , address=45.91.116.5:4297 , piece_id=XEGWKAMHMY3CRSMXGNAL4CDFDEFIVEDQINC7TXKILYIV73QV4UZQ, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:38.912793028 (640h31m38.783876063s) [noise=true , address=77.90.31.14:4661 , piece_id=BPDHJMIUVCABD22C32WNGZZLNGGMWHI46PMQ7MBXBEDYPMTYPRZQ, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:38.942787882 (640h31m38.813870917s) [noise=true , address=77.90.31.19:4841 , piece_id=W4EMUECOZ6NXFBZROVL6W6SI2AADAT6QZMKVZWFUOOJ6RPUHOGQQ, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:39.097452903 (640h31m38.968535948s) [noise=true , address=45.91.116.8:4048 , piece_id=ZN4WX6RP777Z6PHCE6UZOKKCK72YOXTCIHBSGNCWGMVSCC3ZW4KA, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:39.097478132 (640h31m38.968561167s) [seg_pos={0 0}, upload=561245]: Segment reached optimal threshold of 13 pieces.
Jan 1 08:00:39.097913300 (640h31m38.968996335s) [noise=true , address=145.79.227.33:4870 , piece_id=OYMN4GE2TWKCSLFDJ66GJ6UB2ZXDHK7D4CC5A7AYH4NGJCAT66JA, seg_pos={0 0}, upload=561245]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF
Jan 1 08:00:39.097923670 (640h31m38.969006705s) [noise=true , address=145.79.227.33:4870 , piece_id=OYMN4GE2TWKCSLFDJ66GJ6UB2ZXDHK7D4CC5A7AYH4NGJCAT66JA, seg_pos={0 0}, upload=561245]: Long tail cancellation due to optimal threshold
Jan 1 08:00:39.097970158 (640h31m38.969053203s) [noise=true , address=145.79.227.30:4762 , piece_id=WZI2E3HGEQXIMFQBWBZ7PK2W5NDZMNBCKRF3EDB4MSOCA3DGD7JQ, seg_pos={0 0}, upload=561245]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF
Jan 1 08:00:39.097981329 (640h31m38.969064374s) [noise=true , address=145.79.227.30:4762 , piece_id=WZI2E3HGEQXIMFQBWBZ7PK2W5NDZMNBCKRF3EDB4MSOCA3DGD7JQ, seg_pos={0 0}, upload=561245]: Long tail cancellation due to optimal threshold
Jan 1 08:00:39.098151002 (640h31m38.969234048s) [noise=true , address=145.79.227.29:4942 , piece_id=WDSA5DJWK3OVIAOAUCR4I2VTSG4HD4UT7EMV7SECRQHWUPZV2Y4A, seg_pos={0 0}, upload=561245]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF
Jan 1 08:00:39.098160401 (640h31m38.969243436s) [noise=true , address=145.79.227.29:4942 , piece_id=WDSA5DJWK3OVIAOAUCR4I2VTSG4HD4UT7EMV7SECRQHWUPZV2Y4A, seg_pos={0 0}, upload=561245]: Long tail cancellation due to optimal threshold
Jan 1 08:00:39.098182162 (640h31m38.969265207s) [seg_pos={0 0}, upload=561245]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 1 08:00:39.098277884 (640h31m38.96936094s) [upload=561245]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 1 08:00:39.098285188 (640h31m38.969368243s) [upload=561245]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 1 08:00:39.388401257 (640h31m39.259484302s) [upload=561245]: Done uploading segments...
Jan 1 08:00:39.388428308 (640h31m39.259511343s) [upload=561245]: Upload finished. err: <nil>
Duration: 6.684310386s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
20 out of 60 is failed (threshold=3) [X...X.X.XX.XX..X...X.XX.X...X.........X.X........XX.X...X..X] duration:
30.004595378s trace:
https://jaeger.dev.storj.io/trace/14a6f6c583f5aa92
FAIL (2026-01-01 08:01:05 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue, Piece ID: BPDHJMIUVCABD22C32WNGZZLNGGMWHI46PMQ7MBXBEDYPMTYPRZQ) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: ZN4WX6RP777Z6PHCE6UZOKKCK72YOXTCIHBSGNCWGMVSCC3ZW4KA) context deadline exceeded
======LOGS======
Jan 1 08:00:36.336451400 (640h31m36.207534445s) [upload=561245]: Starting upload
Jan 1 08:00:36.336534237 (640h31m36.207617283s) [upload=561245]: Done starting upload
Jan 1 08:00:36.336736032 (640h31m36.207819077s) [upload=561245]: Uploading segments...
Jan 1 08:00:36.336819661 (640h31m36.207902706s) [upload=561245]: Got next segment. Inline: false
Jan 1 08:00:36.336868724 (640h31m36.207951759s) [upload=561245]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 1 08:00:36.336875117 (640h31m36.207958162s) [upload=561245]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 1 08:00:36.580601654 (640h31m36.451684689s) [seg_pos={0 0}, upload=561245]: Begin upload segment...
Jan 1 08:00:36.580652981 (640h31m36.451736026s) [seg_pos={0 0}, upload=561245]: Done begin upload segment.
Jan 1 08:00:36.580707806 (640h31m36.451790851s) [noise=true , address=77.90.31.19:4841 , piece_id=W4EMUECOZ6NXFBZROVL6W6SI2AADAT6QZMKVZWFUOOJ6RPUHOGQQ, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.580843394 (640h31m36.451926429s) [noise=true , address=145.79.227.21:4475 , piece_id=H6RSWRNNW7U5XNKC27OEKZ5ILDOX73OCOZ6N2L35RKZ6TSZPHNFQ, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.581549899 (640h31m36.452632944s) [noise=true , address=145.79.227.29:4942 , piece_id=WDSA5DJWK3OVIAOAUCR4I2VTSG4HD4UT7EMV7SECRQHWUPZV2Y4A, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.582165562 (640h31m36.453248597s) [noise=true , address=145.79.227.48:4302 , piece_id=NJ3UEHC2WBCH7KJO3WZVHXKP3FDLHXKQULVEL7YMNMBU4OIKZUOA, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.582576053 (640h31m36.453659088s) [noise=true , address=145.79.227.30:4762 , piece_id=WZI2E3HGEQXIMFQBWBZ7PK2W5NDZMNBCKRF3EDB4MSOCA3DGD7JQ, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.583033354 (640h31m36.454116399s) [noise=true , address=145.79.227.18:4007 , piece_id=G4NXSCJCZ6LN2RLQSFGQX4XKLNPZFFZUOFROM3SFMOGFUYLWRHNQ, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.583471328 (640h31m36.454554373s) [noise=true , address=45.91.116.5:4297 , piece_id=XEGWKAMHMY3CRSMXGNAL4CDFDEFIVEDQINC7TXKILYIV73QV4UZQ, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.584935826 (640h31m36.456018871s) [noise=true , address=77.90.31.14:4661 , piece_id=BPDHJMIUVCABD22C32WNGZZLNGGMWHI46PMQ7MBXBEDYPMTYPRZQ, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.585046487 (640h31m36.456129522s) [noise=true , address=145.79.227.14:4840 , piece_id=54YIHG6H34PFOQO4OU3MV74CZUCFY5N2OCATSIOCMD453XOUANWQ, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.586542315 (640h31m36.45762535s) [noise=true , address=145.79.227.45:4230 , piece_id=6S3NFOZFIRQUE3T3PEZLZLMNVIOK33JUSYVH5HVWEJ3RMSDKAFXQ, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.586809845 (640h31m36.45789288s) [noise=true , address=145.79.227.33:4870 , piece_id=OYMN4GE2TWKCSLFDJ66GJ6UB2ZXDHK7D4CC5A7AYH4NGJCAT66JA, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.588268963 (640h31m36.459351998s) [noise=true , address=145.79.227.24:4581 , piece_id=DRKKS2N35E7ORYSCARSNHBNKN3DTIGV5STIISCK7THZW76CDVO3A, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.590930822 (640h31m36.462013867s) [noise=true , address=145.79.227.38:4977 , piece_id=J3YUHNPAJEX5D76LVFHZBAP73HSG5IN7ZI5VD7IX4ZVMVGHSAHAA, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.594801081 (640h31m36.465884116s) [noise=true , address=145.79.227.52:4447 , piece_id=R7LXPRI2SWX2PAAALMPFOHP3JNJAFJ5QLJINAFKVK7H6PEE2TTGA, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.598974989 (640h31m36.470058024s) [noise=true , address=145.79.227.9:4367 , piece_id=JGIZMNV72PTWNVAKBO2CQMG6VGXJA6YUJABLSTORA3ZMDO6HYHRQ, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:36.600900144 (640h31m36.47198319s) [noise=true , address=45.91.116.8:4048 , piece_id=ZN4WX6RP777Z6PHCE6UZOKKCK72YOXTCIHBSGNCWGMVSCC3ZW4KA, seg_pos={0 0}, upload=561245]: Uploading piece...
Jan 1 08:00:38.299065741 (640h31m38.170148776s) [upload=561245]: Next returned nil segment
Jan 1 08:00:38.299077214 (640h31m38.170160259s) [upload=561245]: Waiting for error group managing segments...
Jan 1 08:00:38.586542564 (640h31m38.45762562s) [noise=true , address=145.79.227.21:4475 , piece_id=H6RSWRNNW7U5XNKC27OEKZ5ILDOX73OCOZ6N2L35RKZ6TSZPHNFQ, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:38.587267896 (640h31m38.458350941s) [noise=true , address=145.79.227.52:4447 , piece_id=R7LXPRI2SWX2PAAALMPFOHP3JNJAFJ5QLJINAFKVK7H6PEE2TTGA, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:38.587822492 (640h31m38.458905537s) [noise=true , address=145.79.227.18:4007 , piece_id=G4NXSCJCZ6LN2RLQSFGQX4XKLNPZFFZUOFROM3SFMOGFUYLWRHNQ, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:38.600692919 (640h31m38.471775964s) [noise=true , address=145.79.227.45:4230 , piece_id=6S3NFOZFIRQUE3T3PEZLZLMNVIOK33JUSYVH5HVWEJ3RMSDKAFXQ, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:38.601771583 (640h31m38.472854628s) [noise=true , address=145.79.227.48:4302 , piece_id=NJ3UEHC2WBCH7KJO3WZVHXKP3FDLHXKQULVEL7YMNMBU4OIKZUOA, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:38.606091358 (640h31m38.477174403s) [noise=true , address=145.79.227.38:4977 , piece_id=J3YUHNPAJEX5D76LVFHZBAP73HSG5IN7ZI5VD7IX4ZVMVGHSAHAA, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:38.611687373 (640h31m38.482770418s) [noise=true , address=145.79.227.14:4840 , piece_id=54YIHG6H34PFOQO4OU3MV74CZUCFY5N2OCATSIOCMD453XOUANWQ, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:38.614887266 (640h31m38.485970312s) [noise=true , address=145.79.227.24:4581 , piece_id=DRKKS2N35E7ORYSCARSNHBNKN3DTIGV5STIISCK7THZW76CDVO3A, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:38.626493386 (640h31m38.497576431s) [noise=true , address=145.79.227.9:4367 , piece_id=JGIZMNV72PTWNVAKBO2CQMG6VGXJA6YUJABLSTORA3ZMDO6HYHRQ, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:38.910948826 (640h31m38.782031871s) [noise=true , address=45.91.116.5:4297 , piece_id=XEGWKAMHMY3CRSMXGNAL4CDFDEFIVEDQINC7TXKILYIV73QV4UZQ, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:38.912793028 (640h31m38.783876063s) [noise=true , address=77.90.31.14:4661 , piece_id=BPDHJMIUVCABD22C32WNGZZLNGGMWHI46PMQ7MBXBEDYPMTYPRZQ, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:38.942787882 (640h31m38.813870917s) [noise=true , address=77.90.31.19:4841 , piece_id=W4EMUECOZ6NXFBZROVL6W6SI2AADAT6QZMKVZWFUOOJ6RPUHOGQQ, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:39.097452903 (640h31m38.968535948s) [noise=true , address=45.91.116.8:4048 , piece_id=ZN4WX6RP777Z6PHCE6UZOKKCK72YOXTCIHBSGNCWGMVSCC3ZW4KA, seg_pos={0 0}, upload=561245]: Done uploading piece. err: <nil>
Jan 1 08:00:39.097478132 (640h31m38.968561167s) [seg_pos={0 0}, upload=561245]: Segment reached optimal threshold of 13 pieces.
Jan 1 08:00:39.097913300 (640h31m38.968996335s) [noise=true , address=145.79.227.33:4870 , piece_id=OYMN4GE2TWKCSLFDJ66GJ6UB2ZXDHK7D4CC5A7AYH4NGJCAT66JA, seg_pos={0 0}, upload=561245]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF
Jan 1 08:00:39.097923670 (640h31m38.969006705s) [noise=true , address=145.79.227.33:4870 , piece_id=OYMN4GE2TWKCSLFDJ66GJ6UB2ZXDHK7D4CC5A7AYH4NGJCAT66JA, seg_pos={0 0}, upload=561245]: Long tail cancellation due to optimal threshold
Jan 1 08:00:39.097970158 (640h31m38.969053203s) [noise=true , address=145.79.227.30:4762 , piece_id=WZI2E3HGEQXIMFQBWBZ7PK2W5NDZMNBCKRF3EDB4MSOCA3DGD7JQ, seg_pos={0 0}, upload=561245]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF
Jan 1 08:00:39.097981329 (640h31m38.969064374s) [noise=true , address=145.79.227.30:4762 , piece_id=WZI2E3HGEQXIMFQBWBZ7PK2W5NDZMNBCKRF3EDB4MSOCA3DGD7JQ, seg_pos={0 0}, upload=561245]: Long tail cancellation due to optimal threshold
Jan 1 08:00:39.098151002 (640h31m38.969234048s) [noise=true , address=145.79.227.29:4942 , piece_id=WDSA5DJWK3OVIAOAUCR4I2VTSG4HD4UT7EMV7SECRQHWUPZV2Y4A, seg_pos={0 0}, upload=561245]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7): protocol: expected piece hash; context canceled; EOF
Jan 1 08:00:39.098160401 (640h31m38.969243436s) [noise=true , address=145.79.227.29:4942 , piece_id=WDSA5DJWK3OVIAOAUCR4I2VTSG4HD4UT7EMV7SECRQHWUPZV2Y4A, seg_pos={0 0}, upload=561245]: Long tail cancellation due to optimal threshold
Jan 1 08:00:39.098182162 (640h31m38.969265207s) [seg_pos={0 0}, upload=561245]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 1 08:00:39.098277884 (640h31m38.96936094s) [upload=561245]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 1 08:00:39.098285188 (640h31m38.969368243s) [upload=561245]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 1 08:00:39.388401257 (640h31m39.259484302s) [upload=561245]: Done uploading segments...
Jan 1 08:00:39.388428308 (640h31m39.259511343s) [upload=561245]: Upload finished. err: <nil>
Duration: 30.004622992s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
16.571187456s trace:
https://jaeger.dev.storj.io/trace/44e9d74e3b154e8e
FAIL (2026-01-01 08:00:59 UTC)
orbiter warning: context deadline exceeded
Duration: 16.57121627s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
36 out of 60 is failed (threshold=6) [....XX.XX.X.XXXXX.XX..XXXXX.XXXX..XX.XXX.XX...X.X..X...XXXXX] duration:
5.001701475s trace:
https://jaeger.dev.storj.io/trace/3d91cbfc7f1e2a47
FAIL (2026-01-01 08:00:37 UTC)
orbiter warning: context deadline exceeded
Duration: 5.0017244s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
14.292263756s trace:
https://jaeger.dev.storj.io/trace/44e9d74e3b154e8e
FAIL (2026-01-01 08:00:36 UTC)
orbiter warning: context deadline exceeded
Duration: 14.292293592s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
6 out of 60 is failed (threshold=3) [....................X.....X..XX.........................XX..] duration:
39.932278014s trace:
https://jaeger.dev.storj.io/trace/10d96081b66cd020
FAIL (2026-01-01 08:00:35 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
73329s) [noise=true , address=145.79.227.40:4066 , piece_id=ABZ7356MEBO4XPYAKFSS4EQJYJN4CWNJR3C6NXFWLAEO65KL3J5Q, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.722330803 (640h29m1.593413848s) [noise=true , address=145.79.227.9:4391 , piece_id=XS6HYQTJO355YLXEIDBKHTJ44UAT5UBKHDUCFA744X6TBPGYN5YQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.722848709 (640h29m1.593931754s) [noise=true , address=77.90.31.12:4606 , piece_id=K4E5ECNMOMKEIFHZ3EZ2QSQHZ5IYP3CAD5DA6DBQLM4JYMOYU7MA, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.723452529 (640h29m1.594535574s) [noise=true , address=193.149.164.143:4694 , piece_id=EUO4UWMROIFFAJECCFRJBQF4XV25VDKGKX3UF2PTUMTE5JBGD67Q, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.723730728 (640h29m1.594813773s) [noise=true , address=45.91.116.5:4320 , piece_id=FOFLP7277VUUPTRZZHQLNVJ3YVYBELJTUYL7PTT7FJNN7J7QLS2A, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.724307808 (640h29m1.595390853s) [noise=true , address=64.49.13.169:4705 , piece_id=2P3JANSNK64726X7IYJOIQOKAQ4P4LJ5A2GVPX6UXOWUDIYRPZMA, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.724847144 (640h29m1.595930189s) [noise=true , address=64.49.13.137:4494 , piece_id=GCFOH6XRKDEGNG6TFPCMHG5RUCP2LUFS3XEOPNF42VYTGS63ZHQQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.724970960 (640h29m1.596054005s) [noise=true , address=77.90.31.15:4725 , piece_id=LLGZBPL7CHDI2TAY2SYMUMC264USSUQRVFR5P4HTLBWPBOTSUSOA, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.725289036 (640h29m1.596372071s) [noise=true , address=145.79.227.49:4371 , piece_id=RGZZMDFFVVSIQJ2ZEUSK275SBWMGV75WGNDLY6LJ2TNRPUSY7RMQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.725518012 (640h29m1.596601047s) [noise=true , address=77.90.31.13:4649 , piece_id=YRKZDAT5O2USQM4SO5GRZ45SX7MA3U52KN2CRGYPX5WJ4HIKOJBA, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.725902264 (640h29m1.596985309s) [noise=true , address=45.91.116.12:4238 , piece_id=R5LPEYTYC3XB76SSMDVHPZORAWB2P3S4TETMYV23NJI4UXYUYCEQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.726176987 (640h29m1.597260032s) [noise=true , address=145.79.227.35:4497 , piece_id=YDNRFAJVXHRJD5QHICHR6CGF6K5ARZTLHOUSRLV6JAYU3ZSWEY3A, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.726843697 (640h29m1.597926742s) [noise=true , address=77.90.31.19:4846 , piece_id=2XJSN2WTR2BAVKDJZXEMY6SRX7KLOU4SVGYH7TNJIJTPW723DEHQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.727470170 (640h29m1.598553225s) [noise=true , address=145.79.227.30:4784 , piece_id=HA6Z6M3HASDLIDP2PHS7MUDOHBF6T6ZNPTABXF5PYXR2HFYSBOPQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:01.728088075 (640h29m1.59917112s) [noise=true , address=145.79.227.39:4032 , piece_id=Z7BQPDXOVQ6AEKACBB2DJFI652AT6UIHXUB2HPUJFMK53EF73AWQ, seg_pos={0 0}, upload=561203]: Uploading piece...
Jan 1 07:58:03.152355605 (640h29m3.02343864s) [upload=561203]: Next returned nil segment
Jan 1 07:58:03.152371765 (640h29m3.023454811s) [upload=561203]: Waiting for error group managing segments...
Jan 1 07:58:03.563991563 (640h29m3.435074608s) [noise=true , address=145.79.227.9:4391 , piece_id=XS6HYQTJO355YLXEIDBKHTJ44UAT5UBKHDUCFA744X6TBPGYN5YQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:03.652285026 (640h29m3.523368071s) [noise=true , address=145.79.227.49:4371 , piece_id=RGZZMDFFVVSIQJ2ZEUSK275SBWMGV75WGNDLY6LJ2TNRPUSY7RMQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:03.671359306 (640h29m3.542442351s) [noise=true , address=145.79.227.39:4032 , piece_id=Z7BQPDXOVQ6AEKACBB2DJFI652AT6UIHXUB2HPUJFMK53EF73AWQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:03.774991565 (640h29m3.64607462s) [noise=true , address=77.90.31.19:4846 , piece_id=2XJSN2WTR2BAVKDJZXEMY6SRX7KLOU4SVGYH7TNJIJTPW723DEHQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:03.870781221 (640h29m3.741864266s) [noise=true , address=77.90.31.12:4606 , piece_id=K4E5ECNMOMKEIFHZ3EZ2QSQHZ5IYP3CAD5DA6DBQLM4JYMOYU7MA, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:03.878179488 (640h29m3.749262533s) [noise=true , address=77.90.31.13:4649 , piece_id=YRKZDAT5O2USQM4SO5GRZ45SX7MA3U52KN2CRGYPX5WJ4HIKOJBA, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:03.977331467 (640h29m3.848414512s) [noise=true , address=77.90.31.15:4725 , piece_id=LLGZBPL7CHDI2TAY2SYMUMC264USSUQRVFR5P4HTLBWPBOTSUSOA, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.224768738 (640h29m4.095851783s) [noise=true , address=45.91.116.12:4238 , piece_id=R5LPEYTYC3XB76SSMDVHPZORAWB2P3S4TETMYV23NJI4UXYUYCEQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.457462948 (640h29m4.328545994s) [noise=true , address=45.91.116.5:4320 , piece_id=FOFLP7277VUUPTRZZHQLNVJ3YVYBELJTUYL7PTT7FJNN7J7QLS2A, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.488291199 (640h29m4.359374234s) [noise=true , address=145.79.227.35:4497 , piece_id=YDNRFAJVXHRJD5QHICHR6CGF6K5ARZTLHOUSRLV6JAYU3ZSWEY3A, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.522541837 (640h29m4.393624882s) [noise=true , address=145.79.227.43:4171 , piece_id=KYGE2RFI6O6DUFWVGJV2IT5VE3U43WJ4ODNQMTZFQH2DAADFC6BQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.525844266 (640h29m4.396927321s) [noise=true , address=145.79.227.33:4886 , piece_id=CYGCX4TM2ZFPCJYVDF52PDNR2OCNZ2NUSUUOUMUHYGCADWWFDHVQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.539045983 (640h29m4.410129028s) [noise=true , address=145.79.227.34:4924 , piece_id=QUPSGORFPMQVD2IRAK7COO2EE7XDV2KWEBTI2FQ2C25RTAAFNSRA, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.544558039 (640h29m4.415641084s) [noise=true , address=145.79.227.30:4784 , piece_id=HA6Z6M3HASDLIDP2PHS7MUDOHBF6T6ZNPTABXF5PYXR2HFYSBOPQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: <nil>
Jan 1 07:58:04.544582956 (640h29m4.415665991s) [seg_pos={0 0}, upload=561203]: Segment reached optimal threshold of 14 pieces.
Jan 1 07:58:04.544811462 (640h29m4.415894527s) [noise=true , address=193.149.164.143:4694 , piece_id=EUO4UWMROIFFAJECCFRJBQF4XV25VDKGKX3UF2PTUMTE5JBGD67Q, seg_pos={0 0}, upload=561203]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ATtVD3XyFxzEqwQpSF4t1MWygqEa4Y1CkNj1s43eud8UtuSmx): protocol: expected piece hash; context canceled; EOF
Jan 1 07:58:04.544821240 (640h29m4.415904275s) [noise=true , address=193.149.164.143:4694 , piece_id=EUO4UWMROIFFAJECCFRJBQF4XV25VDKGKX3UF2PTUMTE5JBGD67Q, seg_pos={0 0}, upload=561203]: Long tail cancellation due to optimal threshold
Jan 1 07:58:04.545116732 (640h29m4.416199767s) [noise=true , address=64.49.13.169:4705 , piece_id=2P3JANSNK64726X7IYJOIQOKAQ4P4LJ5A2GVPX6UXOWUDIYRPZMA, seg_pos={0 0}, upload=561203]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124uqzVwsF5TM4CwUhkzfJvgxd71jZXgXuFNwDLRW1vrSXbwLsr): protocol: write tcp 188.166.225.170:45312->64.49.13.169:4705: use of closed network connection; write tcp 188.166.225.170:45312->64.49.13.169:4705: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45312->64.49.13.169:4705: use of closed network connection
Jan 1 07:58:04.545123666 (640h29m4.416206711s) [noise=true , address=64.49.13.169:4705 , piece_id=2P3JANSNK64726X7IYJOIQOKAQ4P4LJ5A2GVPX6UXOWUDIYRPZMA, seg_pos={0 0}, upload=561203]: Long tail cancellation due to optimal threshold
Jan 1 07:58:04.545303088 (640h29m4.416386123s) [noise=true , address=145.79.227.40:4066 , piece_id=ABZ7356MEBO4XPYAKFSS4EQJYJN4CWNJR3C6NXFWLAEO65KL3J5Q, seg_pos={0 0}, upload=561203]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tRYJFF6VmxGk8nr1nh6UD3Fnq76qu2Tgkq7cWpV6E8GVpsAUC): protocol: expected piece hash; context canceled; EOF
Jan 1 07:58:04.545312055 (640h29m4.4163951s) [noise=true , address=145.79.227.40:4066 , piece_id=ABZ7356MEBO4XPYAKFSS4EQJYJN4CWNJR3C6NXFWLAEO65KL3J5Q, seg_pos={0 0}, upload=561203]: Long tail cancellation due to optimal threshold
Jan 1 07:58:04.545414279 (640h29m4.416497314s) [noise=true , address=64.49.13.137:4494 , piece_id=GCFOH6XRKDEGNG6TFPCMHG5RUCP2LUFS3XEOPNF42VYTGS63ZHQQ, seg_pos={0 0}, upload=561203]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TX66fDVv3qu4yBGx9FNZTiArNeUnqxjMYwb3VGqYCQRe8RwoU): protocol: write tcp 188.166.225.170:35392->64.49.13.137:4494: use of closed network connection; write tcp 188.166.225.170:35392->64.49.13.137:4494: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35392->64.49.13.137:4494: use of closed network connection
Jan 1 07:58:04.545422815 (640h29m4.41650585s) [noise=true , address=64.49.13.137:4494 , piece_id=GCFOH6XRKDEGNG6TFPCMHG5RUCP2LUFS3XEOPNF42VYTGS63ZHQQ, seg_pos={0 0}, upload=561203]: Long tail cancellation due to optimal threshold
Jan 1 07:58:04.545439939 (640h29m4.416522974s) [seg_pos={0 0}, upload=561203]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 1 07:58:04.545580406 (640h29m4.416663441s) [upload=561203]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 1 07:58:04.545587800 (640h29m4.416670845s) [upload=561203]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 1 07:58:04.840726371 (640h29m4.711809416s) [upload=561203]: Done uploading segments...
Jan 1 07:58:04.840749646 (640h29m4.711832691s) [upload=561203]: Upload finished. err: <nil>
Duration: 39.932305967s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
35 out of 60 is failed (threshold=6) [.....XX.XX.X.XXXXX.XX..XXXXX.XXXX..XX.XXX.XX...X.X..X...XXXX] duration:
5.000811391s trace:
https://jaeger.dev.storj.io/trace/4169d2b6cc369e12
FAIL (2026-01-01 08:00:00 UTC)
orbiter warning: context deadline exceeded
Duration: 5.000834965s
core
uplink
us-select-1
us1-placement-20
|