Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [X.X.X.X.X....XX.X....X...X.X...XXX..X..X........X...XX......] duration:
6.015054234s trace:
https://jaeger.dev.storj.io/trace/735677880864da8a
FAIL (2025-12-31 19:06:05 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: CU2OUXDQYV6ULLGUUUC464LIWQS45JTPB3CTMNQZC3JMIYH2SH4Q) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: 4PUJ6YDAY364WVULRTHYBMG6MBLFDWKLLIPEVYFWK6HCQF3PC6LQ) context deadline exceeded
======LOGS======
segments...
Dec 31 19:01:11.769976062 (627h32m11.641059107s) [upload=549888]: Got next segment. Inline: false
Dec 31 19:01:11.770023953 (627h32m11.641106998s) [upload=549888]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 19:01:11.770043050 (627h32m11.641126085s) [upload=549888]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 19:01:12.023231878 (627h32m11.894314913s) [seg_pos={0 0}, upload=549888]: Begin upload segment...
Dec 31 19:01:12.023296881 (627h32m11.894379916s) [seg_pos={0 0}, upload=549888]: Done begin upload segment.
Dec 31 19:01:12.023355753 (627h32m11.894438798s) [noise=true , address=77.90.31.10:4519 , piece_id=C25QYCH2VTZ6PE72D467NNLTBB74MU74XJQ36CQYTWO5TG22HG7A, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.023376493 (627h32m11.894459528s) [noise=true , address=77.90.31.14:4662 , piece_id=CU2OUXDQYV6ULLGUUUC464LIWQS45JTPB3CTMNQZC3JMIYH2SH4Q, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.023952419 (627h32m11.895035454s) [noise=true , address=45.91.116.11:4119 , piece_id=LDYC6K6PPHDKOWGNEQXU464PFIBYSPWXMJZBEY7ELBRPP5YL57EA, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.024015850 (627h32m11.895098885s) [noise=true , address=77.90.31.16:4736 , piece_id=PYWN35W6PGUC3NMTIROYVLH7UY2TJH3LRJWWEXBXSAXREX5QJX2Q, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.024311754 (627h32m11.895394789s) [noise=true , address=145.79.227.46:4603 , piece_id=RTKSKQ4L63I7MBTLOAMZE36DNSPQ2ZPXV64KRMQPNWCHHODZIYGA, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.024431361 (627h32m11.895514397s) [noise=true , address=145.79.227.6:4259 , piece_id=OQAYR6PIVUKBG2A3OKZBZX3H7ZJFEUR256I3IU6QWYQ4X7PVZGSQ, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.024577730 (627h32m11.895660765s) [noise=true , address=145.79.227.14:4840 , piece_id=QYG4BKZ54RFDZGVXEYNGXRAL4INUYCE4CLPWKCETGL7DSBOKEFKA, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.024765618 (627h32m11.895848653s) [noise=true , address=45.91.116.14:4189 , piece_id=4PUJ6YDAY364WVULRTHYBMG6MBLFDWKLLIPEVYFWK6HCQF3PC6LQ, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.024809993 (627h32m11.895893028s) [noise=true , address=145.79.227.22:4509 , piece_id=WVFB7RFZSVW2M446KJGVIJ4FUGYC7QLEAB6TYY6J2O563KXBFT6A, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.025153747 (627h32m11.896236782s) [noise=true , address=77.90.31.19:4842 , piece_id=ZE2C5YWUKVXYLJTMKQRFYN6MVZ6EUOP4DLU4FEB6D6F3VXII7ZYA, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.025220134 (627h32m11.896303179s) [noise=true , address=45.91.116.6:4974 , piece_id=R5FADLJDMRBYMO5AZL3MR6GUA55O4KTGJ6EWCB725JBHXSCMCYHA, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.025492613 (627h32m11.896575648s) [noise=true , address=77.90.31.18:4805 , piece_id=ZHDEUEKMLONGWMVC2FXD5OGIYUKO6QF5BSORJTSESP4JUJP7V2EQ, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.025786592 (627h32m11.896869627s) [noise=true , address=45.91.116.13:4156 , piece_id=LOJNZHUI2LDUHLQMKM2ZBRNRKVDEX7UXSBUR3WZJNOSCU36GVEUA, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.026165153 (627h32m11.897248198s) [noise=true , address=145.79.227.48:4303 , piece_id=HGCHG7IT4ZBWBYCPULLYDC6M2CAOCNR4PSIH4JU3Y5WP7FOTYUZA, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.026497455 (627h32m11.8975805s) [noise=true , address=45.91.116.12:4228 , piece_id=I5N63CF36AUCN2OGF66OV5GEVOQYQSUR3CS6YLJWWVMNGUL2W3XA, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.026680595 (627h32m11.89776364s) [noise=true , address=77.90.31.21:4878 , piece_id=FX5WINCM4HKZVSKP6MTEKZOMOEWJRJ4J7E4HVGLG3CQSEDJXJLSQ, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:13.730039170 (627h32m13.601122215s) [upload=549888]: Next returned nil segment
Dec 31 19:01:13.730056022 (627h32m13.601139057s) [upload=549888]: Waiting for error group managing segments...
Dec 31 19:01:14.011727154 (627h32m13.882810199s) [noise=true , address=145.79.227.46:4603 , piece_id=RTKSKQ4L63I7MBTLOAMZE36DNSPQ2ZPXV64KRMQPNWCHHODZIYGA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.015213612 (627h32m13.886296647s) [noise=true , address=145.79.227.22:4509 , piece_id=WVFB7RFZSVW2M446KJGVIJ4FUGYC7QLEAB6TYY6J2O563KXBFT6A, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.016168130 (627h32m13.887251165s) [noise=true , address=145.79.227.6:4259 , piece_id=OQAYR6PIVUKBG2A3OKZBZX3H7ZJFEUR256I3IU6QWYQ4X7PVZGSQ, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.067093519 (627h32m13.938176554s) [noise=true , address=145.79.227.14:4840 , piece_id=QYG4BKZ54RFDZGVXEYNGXRAL4INUYCE4CLPWKCETGL7DSBOKEFKA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.094476500 (627h32m13.965559545s) [noise=true , address=145.79.227.48:4303 , piece_id=HGCHG7IT4ZBWBYCPULLYDC6M2CAOCNR4PSIH4JU3Y5WP7FOTYUZA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.197378931 (627h32m14.068461976s) [noise=true , address=77.90.31.14:4662 , piece_id=CU2OUXDQYV6ULLGUUUC464LIWQS45JTPB3CTMNQZC3JMIYH2SH4Q, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.260600915 (627h32m14.13168395s) [noise=true , address=77.90.31.10:4519 , piece_id=C25QYCH2VTZ6PE72D467NNLTBB74MU74XJQ36CQYTWO5TG22HG7A, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.292833670 (627h32m14.163916715s) [noise=true , address=77.90.31.19:4842 , piece_id=ZE2C5YWUKVXYLJTMKQRFYN6MVZ6EUOP4DLU4FEB6D6F3VXII7ZYA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.335927357 (627h32m14.207010392s) [noise=true , address=45.91.116.11:4119 , piece_id=LDYC6K6PPHDKOWGNEQXU464PFIBYSPWXMJZBEY7ELBRPP5YL57EA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.363700461 (627h32m14.234783496s) [noise=true , address=45.91.116.13:4156 , piece_id=LOJNZHUI2LDUHLQMKM2ZBRNRKVDEX7UXSBUR3WZJNOSCU36GVEUA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.375270873 (627h32m14.246353918s) [noise=true , address=77.90.31.21:4878 , piece_id=FX5WINCM4HKZVSKP6MTEKZOMOEWJRJ4J7E4HVGLG3CQSEDJXJLSQ, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.458059915 (627h32m14.32914295s) [noise=true , address=45.91.116.14:4189 , piece_id=4PUJ6YDAY364WVULRTHYBMG6MBLFDWKLLIPEVYFWK6HCQF3PC6LQ, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.577656003 (627h32m14.448739038s) [noise=true , address=77.90.31.18:4805 , piece_id=ZHDEUEKMLONGWMVC2FXD5OGIYUKO6QF5BSORJTSESP4JUJP7V2EQ, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.577676772 (627h32m14.448759807s) [seg_pos={0 0}, upload=549888]: Segment reached optimal threshold of 13 pieces.
Dec 31 19:01:14.578054592 (627h32m14.449137627s) [noise=true , address=45.91.116.12:4228 , piece_id=I5N63CF36AUCN2OGF66OV5GEVOQYQSUR3CS6YLJWWVMNGUL2W3XA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:47206->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:47206->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47206->45.91.116.12:4228: use of closed network connection
Dec 31 19:01:14.578062086 (627h32m14.449145121s) [noise=true , address=45.91.116.12:4228 , piece_id=I5N63CF36AUCN2OGF66OV5GEVOQYQSUR3CS6YLJWWVMNGUL2W3XA, seg_pos={0 0}, upload=549888]: Long tail cancellation due to optimal threshold
Dec 31 19:01:14.578143692 (627h32m14.449226747s) [noise=true , address=77.90.31.16:4736 , piece_id=PYWN35W6PGUC3NMTIROYVLH7UY2TJH3LRJWWEXBXSAXREX5QJX2Q, seg_pos={0 0}, upload=549888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF
Dec 31 19:01:14.578156606 (627h32m14.449239671s) [noise=true , address=77.90.31.16:4736 , piece_id=PYWN35W6PGUC3NMTIROYVLH7UY2TJH3LRJWWEXBXSAXREX5QJX2Q, seg_pos={0 0}, upload=549888]: Long tail cancellation due to optimal threshold
Dec 31 19:01:14.578161215 (627h32m14.44924425s) [noise=true , address=45.91.116.6:4974 , piece_id=R5FADLJDMRBYMO5AZL3MR6GUA55O4KTGJ6EWCB725JBHXSCMCYHA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:48032->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:48032->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48032->45.91.116.6:4974: use of closed network connection
Dec 31 19:01:14.578167326 (627h32m14.449250361s) [noise=true , address=45.91.116.6:4974 , piece_id=R5FADLJDMRBYMO5AZL3MR6GUA55O4KTGJ6EWCB725JBHXSCMCYHA, seg_pos={0 0}, upload=549888]: Long tail cancellation due to optimal threshold
Dec 31 19:01:14.578181263 (627h32m14.449264308s) [seg_pos={0 0}, upload=549888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 19:01:14.578277015 (627h32m14.44936005s) [upload=549888]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 19:01:14.578281534 (627h32m14.449364589s) [upload=549888]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 19:01:14.875461937 (627h32m14.746544982s) [upload=549888]: Done uploading segments...
Dec 31 19:01:14.875484871 (627h32m14.746567916s) [upload=549888]: Upload finished. err: <nil>
Duration: 6.015072528s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..X.........................................................] duration:
3.45632992s trace:
https://jaeger.dev.storj.io/trace/1c0a173ba9d72fc4
FAIL (2025-12-31 19:05:55 UTC)
orbiter warning: context deadline exceeded
Duration: 3.456359287s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [.X.X.X.X.X....XX.X....X...X.X...XXX..X..X........X...XX.....] duration:
23.656927519s trace:
https://jaeger.dev.storj.io/trace/735677880864da8a
FAIL (2025-12-31 19:05:42 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: CU2OUXDQYV6ULLGUUUC464LIWQS45JTPB3CTMNQZC3JMIYH2SH4Q) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: 4PUJ6YDAY364WVULRTHYBMG6MBLFDWKLLIPEVYFWK6HCQF3PC6LQ) context deadline exceeded
======LOGS======
segments...
Dec 31 19:01:11.769976062 (627h32m11.641059107s) [upload=549888]: Got next segment. Inline: false
Dec 31 19:01:11.770023953 (627h32m11.641106998s) [upload=549888]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 19:01:11.770043050 (627h32m11.641126085s) [upload=549888]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 19:01:12.023231878 (627h32m11.894314913s) [seg_pos={0 0}, upload=549888]: Begin upload segment...
Dec 31 19:01:12.023296881 (627h32m11.894379916s) [seg_pos={0 0}, upload=549888]: Done begin upload segment.
Dec 31 19:01:12.023355753 (627h32m11.894438798s) [noise=true , address=77.90.31.10:4519 , piece_id=C25QYCH2VTZ6PE72D467NNLTBB74MU74XJQ36CQYTWO5TG22HG7A, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.023376493 (627h32m11.894459528s) [noise=true , address=77.90.31.14:4662 , piece_id=CU2OUXDQYV6ULLGUUUC464LIWQS45JTPB3CTMNQZC3JMIYH2SH4Q, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.023952419 (627h32m11.895035454s) [noise=true , address=45.91.116.11:4119 , piece_id=LDYC6K6PPHDKOWGNEQXU464PFIBYSPWXMJZBEY7ELBRPP5YL57EA, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.024015850 (627h32m11.895098885s) [noise=true , address=77.90.31.16:4736 , piece_id=PYWN35W6PGUC3NMTIROYVLH7UY2TJH3LRJWWEXBXSAXREX5QJX2Q, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.024311754 (627h32m11.895394789s) [noise=true , address=145.79.227.46:4603 , piece_id=RTKSKQ4L63I7MBTLOAMZE36DNSPQ2ZPXV64KRMQPNWCHHODZIYGA, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.024431361 (627h32m11.895514397s) [noise=true , address=145.79.227.6:4259 , piece_id=OQAYR6PIVUKBG2A3OKZBZX3H7ZJFEUR256I3IU6QWYQ4X7PVZGSQ, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.024577730 (627h32m11.895660765s) [noise=true , address=145.79.227.14:4840 , piece_id=QYG4BKZ54RFDZGVXEYNGXRAL4INUYCE4CLPWKCETGL7DSBOKEFKA, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.024765618 (627h32m11.895848653s) [noise=true , address=45.91.116.14:4189 , piece_id=4PUJ6YDAY364WVULRTHYBMG6MBLFDWKLLIPEVYFWK6HCQF3PC6LQ, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.024809993 (627h32m11.895893028s) [noise=true , address=145.79.227.22:4509 , piece_id=WVFB7RFZSVW2M446KJGVIJ4FUGYC7QLEAB6TYY6J2O563KXBFT6A, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.025153747 (627h32m11.896236782s) [noise=true , address=77.90.31.19:4842 , piece_id=ZE2C5YWUKVXYLJTMKQRFYN6MVZ6EUOP4DLU4FEB6D6F3VXII7ZYA, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.025220134 (627h32m11.896303179s) [noise=true , address=45.91.116.6:4974 , piece_id=R5FADLJDMRBYMO5AZL3MR6GUA55O4KTGJ6EWCB725JBHXSCMCYHA, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.025492613 (627h32m11.896575648s) [noise=true , address=77.90.31.18:4805 , piece_id=ZHDEUEKMLONGWMVC2FXD5OGIYUKO6QF5BSORJTSESP4JUJP7V2EQ, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.025786592 (627h32m11.896869627s) [noise=true , address=45.91.116.13:4156 , piece_id=LOJNZHUI2LDUHLQMKM2ZBRNRKVDEX7UXSBUR3WZJNOSCU36GVEUA, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.026165153 (627h32m11.897248198s) [noise=true , address=145.79.227.48:4303 , piece_id=HGCHG7IT4ZBWBYCPULLYDC6M2CAOCNR4PSIH4JU3Y5WP7FOTYUZA, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.026497455 (627h32m11.8975805s) [noise=true , address=45.91.116.12:4228 , piece_id=I5N63CF36AUCN2OGF66OV5GEVOQYQSUR3CS6YLJWWVMNGUL2W3XA, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.026680595 (627h32m11.89776364s) [noise=true , address=77.90.31.21:4878 , piece_id=FX5WINCM4HKZVSKP6MTEKZOMOEWJRJ4J7E4HVGLG3CQSEDJXJLSQ, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:13.730039170 (627h32m13.601122215s) [upload=549888]: Next returned nil segment
Dec 31 19:01:13.730056022 (627h32m13.601139057s) [upload=549888]: Waiting for error group managing segments...
Dec 31 19:01:14.011727154 (627h32m13.882810199s) [noise=true , address=145.79.227.46:4603 , piece_id=RTKSKQ4L63I7MBTLOAMZE36DNSPQ2ZPXV64KRMQPNWCHHODZIYGA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.015213612 (627h32m13.886296647s) [noise=true , address=145.79.227.22:4509 , piece_id=WVFB7RFZSVW2M446KJGVIJ4FUGYC7QLEAB6TYY6J2O563KXBFT6A, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.016168130 (627h32m13.887251165s) [noise=true , address=145.79.227.6:4259 , piece_id=OQAYR6PIVUKBG2A3OKZBZX3H7ZJFEUR256I3IU6QWYQ4X7PVZGSQ, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.067093519 (627h32m13.938176554s) [noise=true , address=145.79.227.14:4840 , piece_id=QYG4BKZ54RFDZGVXEYNGXRAL4INUYCE4CLPWKCETGL7DSBOKEFKA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.094476500 (627h32m13.965559545s) [noise=true , address=145.79.227.48:4303 , piece_id=HGCHG7IT4ZBWBYCPULLYDC6M2CAOCNR4PSIH4JU3Y5WP7FOTYUZA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.197378931 (627h32m14.068461976s) [noise=true , address=77.90.31.14:4662 , piece_id=CU2OUXDQYV6ULLGUUUC464LIWQS45JTPB3CTMNQZC3JMIYH2SH4Q, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.260600915 (627h32m14.13168395s) [noise=true , address=77.90.31.10:4519 , piece_id=C25QYCH2VTZ6PE72D467NNLTBB74MU74XJQ36CQYTWO5TG22HG7A, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.292833670 (627h32m14.163916715s) [noise=true , address=77.90.31.19:4842 , piece_id=ZE2C5YWUKVXYLJTMKQRFYN6MVZ6EUOP4DLU4FEB6D6F3VXII7ZYA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.335927357 (627h32m14.207010392s) [noise=true , address=45.91.116.11:4119 , piece_id=LDYC6K6PPHDKOWGNEQXU464PFIBYSPWXMJZBEY7ELBRPP5YL57EA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.363700461 (627h32m14.234783496s) [noise=true , address=45.91.116.13:4156 , piece_id=LOJNZHUI2LDUHLQMKM2ZBRNRKVDEX7UXSBUR3WZJNOSCU36GVEUA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.375270873 (627h32m14.246353918s) [noise=true , address=77.90.31.21:4878 , piece_id=FX5WINCM4HKZVSKP6MTEKZOMOEWJRJ4J7E4HVGLG3CQSEDJXJLSQ, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.458059915 (627h32m14.32914295s) [noise=true , address=45.91.116.14:4189 , piece_id=4PUJ6YDAY364WVULRTHYBMG6MBLFDWKLLIPEVYFWK6HCQF3PC6LQ, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.577656003 (627h32m14.448739038s) [noise=true , address=77.90.31.18:4805 , piece_id=ZHDEUEKMLONGWMVC2FXD5OGIYUKO6QF5BSORJTSESP4JUJP7V2EQ, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.577676772 (627h32m14.448759807s) [seg_pos={0 0}, upload=549888]: Segment reached optimal threshold of 13 pieces.
Dec 31 19:01:14.578054592 (627h32m14.449137627s) [noise=true , address=45.91.116.12:4228 , piece_id=I5N63CF36AUCN2OGF66OV5GEVOQYQSUR3CS6YLJWWVMNGUL2W3XA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:47206->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:47206->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47206->45.91.116.12:4228: use of closed network connection
Dec 31 19:01:14.578062086 (627h32m14.449145121s) [noise=true , address=45.91.116.12:4228 , piece_id=I5N63CF36AUCN2OGF66OV5GEVOQYQSUR3CS6YLJWWVMNGUL2W3XA, seg_pos={0 0}, upload=549888]: Long tail cancellation due to optimal threshold
Dec 31 19:01:14.578143692 (627h32m14.449226747s) [noise=true , address=77.90.31.16:4736 , piece_id=PYWN35W6PGUC3NMTIROYVLH7UY2TJH3LRJWWEXBXSAXREX5QJX2Q, seg_pos={0 0}, upload=549888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF
Dec 31 19:01:14.578156606 (627h32m14.449239671s) [noise=true , address=77.90.31.16:4736 , piece_id=PYWN35W6PGUC3NMTIROYVLH7UY2TJH3LRJWWEXBXSAXREX5QJX2Q, seg_pos={0 0}, upload=549888]: Long tail cancellation due to optimal threshold
Dec 31 19:01:14.578161215 (627h32m14.44924425s) [noise=true , address=45.91.116.6:4974 , piece_id=R5FADLJDMRBYMO5AZL3MR6GUA55O4KTGJ6EWCB725JBHXSCMCYHA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:48032->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:48032->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48032->45.91.116.6:4974: use of closed network connection
Dec 31 19:01:14.578167326 (627h32m14.449250361s) [noise=true , address=45.91.116.6:4974 , piece_id=R5FADLJDMRBYMO5AZL3MR6GUA55O4KTGJ6EWCB725JBHXSCMCYHA, seg_pos={0 0}, upload=549888]: Long tail cancellation due to optimal threshold
Dec 31 19:01:14.578181263 (627h32m14.449264308s) [seg_pos={0 0}, upload=549888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 19:01:14.578277015 (627h32m14.44936005s) [upload=549888]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 19:01:14.578281534 (627h32m14.449364589s) [upload=549888]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 19:01:14.875461937 (627h32m14.746544982s) [upload=549888]: Done uploading segments...
Dec 31 19:01:14.875484871 (627h32m14.746567916s) [upload=549888]: Upload finished. err: <nil>
Duration: 23.656951434s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...X........................................................] duration:
3.528341552s trace:
https://jaeger.dev.storj.io/trace/1c0a173ba9d72fc4
FAIL (2025-12-31 19:05:29 UTC)
orbiter warning: context deadline exceeded
Duration: 3.528364174s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [........................X..................X.X..............] duration:
24.196332604s trace:
https://jaeger.dev.storj.io/trace/3fb4961d520cfd75
FAIL (2025-12-31 19:05:27 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12bYqoLXPqxPwKHkGRWx6RSpY8KSnS7QeKafAmtEpGJ5ThmJcDD, Piece ID: ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A) context deadline exceeded; piecestore: (Node ID: 1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, Piece ID: EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ) context deadline exceeded; piecestore: (Node ID: 1zF1SqsUdx5ikfqnG4SfeFukn8ZgX56sC7DGEPtPxZqEicws6o, Piece ID: PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA) context deadline exceeded
======LOGS======
, address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.958820533 (627h24m29.829903569s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.959307350 (627h24m29.830390386s) [noise=true , address=145.79.227.49:4349 , piece_id=R2IQZU7VSWQRVSJITAEMDNNJIPSJA3LBGC2X27AZ7GNZSN4XW2TA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.959608194 (627h24m29.830691229s) [noise=true , address=145.79.227.24:4588 , piece_id=5DP6OWR4UURMV22UA73ZIWUITQCEERVA6KE2K3T6TVDXAV2SO65A, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.959886383 (627h24m29.830969429s) [noise=true , address=64.49.13.26:4945 , piece_id=UMBXDB3DCEJERPVVINHZPPLJ7GKLDZ4PCZKBEOQD5RY5MB7LPCJQ, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.960144104 (627h24m29.831227139s) [noise=true , address=45.91.116.6:4008 , piece_id=64B6RIG5ANWZEVEMNU5HGC6X6E7EHBAIPDJN77UT6RQYYXRGVRWA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.960436030 (627h24m29.831519065s) [noise=true , address=145.79.227.47:4289 , piece_id=GMAPJV5LDECT7ZKYL23XWLDDTYHRAREVFZ2PEIH6OVJNTHUTTAHQ, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.960723447 (627h24m29.831806482s) [noise=true , address=145.79.227.42:4134 , piece_id=SWUYLD4EMMIYVLSWKSO3G3VOQ32ZGAI7NFX7KOURSIN3X6IZ4YJQ, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.960995856 (627h24m29.832078891s) [noise=true , address=145.79.227.40:4084 , piece_id=FD4PW56PCRS3QYHNXA6UQ3SXW2HNFIV3W53XFKMCQMAHDF542N3A, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.961374016 (627h24m29.832457051s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.961713552 (627h24m29.832796587s) [noise=true , address=64.49.13.137:4501 , piece_id=JBYIIB5EY4R2PJULBSLUFDQ3CMMYYIZHTS4BIX7GV5F2KY5OZC6A, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.962169091 (627h24m29.833252136s) [noise=true , address=145.79.227.2:4132 , piece_id=H2G6OZT52HEDBWWOSPU3VS5XIUT64XP5BIO2U7VAX5LNARERBQFA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.962533905 (627h24m29.83361695s) [noise=true , address=145.79.227.18:4027 , piece_id=RVLZBSJIG7UIDS6ITMGFQZJSJEKKDG4G6RFWYLXPUL7AEDZRZHMA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.963210673 (627h24m29.834293708s) [noise=true , address=145.79.227.35:4502 , piece_id=JDBTVMAGPR7M2TPMWJKMDQ562USDDYYPYFQMGEJSTNPARS6NAD5A, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.963755381 (627h24m29.834838426s) [noise=true , address=64.49.13.27:4957 , piece_id=ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.964023922 (627h24m29.835106957s) [noise=true , address=77.90.31.15:4730 , piece_id=PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.964592434 (627h24m29.835675469s) [noise=true , address=77.90.31.16:4753 , piece_id=EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.965227163 (627h24m29.836310208s) [noise=true , address=145.79.227.30:4770 , piece_id=TRAIDZYLZCU5UGHDYAJREYUELZQ5H5AFU4HHPYQFFKPXRZ5OJAQQ, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:31.410300275 (627h24m31.28138331s) [upload=549775]: Next returned nil segment
Dec 31 18:53:31.410315224 (627h24m31.281398259s) [upload=549775]: Waiting for error group managing segments...
Dec 31 18:53:31.788440578 (627h24m31.659523613s) [noise=true , address=64.49.13.26:4945 , piece_id=UMBXDB3DCEJERPVVINHZPPLJ7GKLDZ4PCZKBEOQD5RY5MB7LPCJQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.858179480 (627h24m31.729262526s) [noise=true , address=145.79.227.42:4134 , piece_id=SWUYLD4EMMIYVLSWKSO3G3VOQ32ZGAI7NFX7KOURSIN3X6IZ4YJQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.887866078 (627h24m31.758949113s) [noise=true , address=145.79.227.24:4588 , piece_id=5DP6OWR4UURMV22UA73ZIWUITQCEERVA6KE2K3T6TVDXAV2SO65A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.918774252 (627h24m31.789857297s) [noise=true , address=145.79.227.18:4027 , piece_id=RVLZBSJIG7UIDS6ITMGFQZJSJEKKDG4G6RFWYLXPUL7AEDZRZHMA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.922490298 (627h24m31.793573333s) [noise=true , address=145.79.227.35:4502 , piece_id=JDBTVMAGPR7M2TPMWJKMDQ562USDDYYPYFQMGEJSTNPARS6NAD5A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.924120612 (627h24m31.795203647s) [noise=true , address=145.79.227.2:4132 , piece_id=H2G6OZT52HEDBWWOSPU3VS5XIUT64XP5BIO2U7VAX5LNARERBQFA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.971450444 (627h24m31.842533479s) [noise=true , address=145.79.227.47:4289 , piece_id=GMAPJV5LDECT7ZKYL23XWLDDTYHRAREVFZ2PEIH6OVJNTHUTTAHQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.973634513 (627h24m31.844717548s) [noise=true , address=64.49.13.27:4957 , piece_id=ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.984147871 (627h24m31.855230906s) [noise=true , address=64.49.13.137:4501 , piece_id=JBYIIB5EY4R2PJULBSLUFDQ3CMMYYIZHTS4BIX7GV5F2KY5OZC6A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.002802704 (627h24m31.873885749s) [noise=true , address=77.90.31.15:4730 , piece_id=PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.113876665 (627h24m31.9849597s) [noise=true , address=77.90.31.16:4753 , piece_id=EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.307710471 (627h24m32.178793506s) [noise=true , address=45.91.116.6:4008 , piece_id=64B6RIG5ANWZEVEMNU5HGC6X6E7EHBAIPDJN77UT6RQYYXRGVRWA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.740289560 (627h24m32.611372605s) [noise=true , address=145.79.227.30:4770 , piece_id=TRAIDZYLZCU5UGHDYAJREYUELZQ5H5AFU4HHPYQFFKPXRZ5OJAQQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.744374660 (627h24m32.615457715s) [noise=true , address=145.79.227.49:4349 , piece_id=R2IQZU7VSWQRVSJITAEMDNNJIPSJA3LBGC2X27AZ7GNZSN4XW2TA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.744399677 (627h24m32.615482712s) [seg_pos={0 0}, upload=549775]: Segment reached optimal threshold of 14 pieces.
Dec 31 18:53:32.744624495 (627h24m32.61570753s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA): protocol: write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection; write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection
Dec 31 18:53:32.744633362 (627h24m32.615716397s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold
Dec 31 18:53:32.744938423 (627h24m32.616021468s) [noise=true , address=145.79.227.40:4084 , piece_id=FD4PW56PCRS3QYHNXA6UQ3SXW2HNFIV3W53XFKMCQMAHDF542N3A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.745025308 (627h24m32.616108343s) [noise=true , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gh7eXJMeRFyjBP2e9ZyTQVVE1mMoMFfSkQ9ssfxJyqG63C4gP): protocol: expected piece hash; context canceled; EOF
Dec 31 18:53:32.745035146 (627h24m32.616118181s) [noise=true , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold
Dec 31 18:53:32.745291255 (627h24m32.61637429s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu): protocol: expected piece hash; context canceled; EOF
Dec 31 18:53:32.745299210 (627h24m32.616382245s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold
Dec 31 18:53:32.745319939 (627h24m32.616402985s) [seg_pos={0 0}, upload=549775]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil>
Dec 31 18:53:32.745400913 (627h24m32.616483948s) [upload=549775]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 18:53:32.745406434 (627h24m32.616489479s) [upload=549775]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 18:53:33.044217872 (627h24m32.915300907s) [upload=549775]: Done uploading segments...
Dec 31 18:53:33.044236999 (627h24m32.915320034s) [upload=549775]: Upload finished. err: <nil>
Duration: 24.196383129s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
6.214567877s trace:
https://jaeger.dev.storj.io/trace/9459b27b9ebbd
FAIL (2025-12-31 19:04:51 UTC)
orbiter warning: context deadline exceeded
Duration: 6.21459594s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
20 out of 60 is failed (threshold=3) [X.X.X.X.X.X....XX.X....X...X.X...XXX..X..X........X...XX....] duration:
5.795739108s trace:
https://jaeger.dev.storj.io/trace/735677880864da8a
FAIL (2025-12-31 19:04:39 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: CU2OUXDQYV6ULLGUUUC464LIWQS45JTPB3CTMNQZC3JMIYH2SH4Q) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: 4PUJ6YDAY364WVULRTHYBMG6MBLFDWKLLIPEVYFWK6HCQF3PC6LQ) context deadline exceeded
======LOGS======
segments...
Dec 31 19:01:11.769976062 (627h32m11.641059107s) [upload=549888]: Got next segment. Inline: false
Dec 31 19:01:11.770023953 (627h32m11.641106998s) [upload=549888]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 19:01:11.770043050 (627h32m11.641126085s) [upload=549888]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 19:01:12.023231878 (627h32m11.894314913s) [seg_pos={0 0}, upload=549888]: Begin upload segment...
Dec 31 19:01:12.023296881 (627h32m11.894379916s) [seg_pos={0 0}, upload=549888]: Done begin upload segment.
Dec 31 19:01:12.023355753 (627h32m11.894438798s) [noise=true , address=77.90.31.10:4519 , piece_id=C25QYCH2VTZ6PE72D467NNLTBB74MU74XJQ36CQYTWO5TG22HG7A, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.023376493 (627h32m11.894459528s) [noise=true , address=77.90.31.14:4662 , piece_id=CU2OUXDQYV6ULLGUUUC464LIWQS45JTPB3CTMNQZC3JMIYH2SH4Q, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.023952419 (627h32m11.895035454s) [noise=true , address=45.91.116.11:4119 , piece_id=LDYC6K6PPHDKOWGNEQXU464PFIBYSPWXMJZBEY7ELBRPP5YL57EA, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.024015850 (627h32m11.895098885s) [noise=true , address=77.90.31.16:4736 , piece_id=PYWN35W6PGUC3NMTIROYVLH7UY2TJH3LRJWWEXBXSAXREX5QJX2Q, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.024311754 (627h32m11.895394789s) [noise=true , address=145.79.227.46:4603 , piece_id=RTKSKQ4L63I7MBTLOAMZE36DNSPQ2ZPXV64KRMQPNWCHHODZIYGA, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.024431361 (627h32m11.895514397s) [noise=true , address=145.79.227.6:4259 , piece_id=OQAYR6PIVUKBG2A3OKZBZX3H7ZJFEUR256I3IU6QWYQ4X7PVZGSQ, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.024577730 (627h32m11.895660765s) [noise=true , address=145.79.227.14:4840 , piece_id=QYG4BKZ54RFDZGVXEYNGXRAL4INUYCE4CLPWKCETGL7DSBOKEFKA, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.024765618 (627h32m11.895848653s) [noise=true , address=45.91.116.14:4189 , piece_id=4PUJ6YDAY364WVULRTHYBMG6MBLFDWKLLIPEVYFWK6HCQF3PC6LQ, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.024809993 (627h32m11.895893028s) [noise=true , address=145.79.227.22:4509 , piece_id=WVFB7RFZSVW2M446KJGVIJ4FUGYC7QLEAB6TYY6J2O563KXBFT6A, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.025153747 (627h32m11.896236782s) [noise=true , address=77.90.31.19:4842 , piece_id=ZE2C5YWUKVXYLJTMKQRFYN6MVZ6EUOP4DLU4FEB6D6F3VXII7ZYA, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.025220134 (627h32m11.896303179s) [noise=true , address=45.91.116.6:4974 , piece_id=R5FADLJDMRBYMO5AZL3MR6GUA55O4KTGJ6EWCB725JBHXSCMCYHA, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.025492613 (627h32m11.896575648s) [noise=true , address=77.90.31.18:4805 , piece_id=ZHDEUEKMLONGWMVC2FXD5OGIYUKO6QF5BSORJTSESP4JUJP7V2EQ, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.025786592 (627h32m11.896869627s) [noise=true , address=45.91.116.13:4156 , piece_id=LOJNZHUI2LDUHLQMKM2ZBRNRKVDEX7UXSBUR3WZJNOSCU36GVEUA, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.026165153 (627h32m11.897248198s) [noise=true , address=145.79.227.48:4303 , piece_id=HGCHG7IT4ZBWBYCPULLYDC6M2CAOCNR4PSIH4JU3Y5WP7FOTYUZA, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.026497455 (627h32m11.8975805s) [noise=true , address=45.91.116.12:4228 , piece_id=I5N63CF36AUCN2OGF66OV5GEVOQYQSUR3CS6YLJWWVMNGUL2W3XA, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.026680595 (627h32m11.89776364s) [noise=true , address=77.90.31.21:4878 , piece_id=FX5WINCM4HKZVSKP6MTEKZOMOEWJRJ4J7E4HVGLG3CQSEDJXJLSQ, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:13.730039170 (627h32m13.601122215s) [upload=549888]: Next returned nil segment
Dec 31 19:01:13.730056022 (627h32m13.601139057s) [upload=549888]: Waiting for error group managing segments...
Dec 31 19:01:14.011727154 (627h32m13.882810199s) [noise=true , address=145.79.227.46:4603 , piece_id=RTKSKQ4L63I7MBTLOAMZE36DNSPQ2ZPXV64KRMQPNWCHHODZIYGA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.015213612 (627h32m13.886296647s) [noise=true , address=145.79.227.22:4509 , piece_id=WVFB7RFZSVW2M446KJGVIJ4FUGYC7QLEAB6TYY6J2O563KXBFT6A, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.016168130 (627h32m13.887251165s) [noise=true , address=145.79.227.6:4259 , piece_id=OQAYR6PIVUKBG2A3OKZBZX3H7ZJFEUR256I3IU6QWYQ4X7PVZGSQ, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.067093519 (627h32m13.938176554s) [noise=true , address=145.79.227.14:4840 , piece_id=QYG4BKZ54RFDZGVXEYNGXRAL4INUYCE4CLPWKCETGL7DSBOKEFKA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.094476500 (627h32m13.965559545s) [noise=true , address=145.79.227.48:4303 , piece_id=HGCHG7IT4ZBWBYCPULLYDC6M2CAOCNR4PSIH4JU3Y5WP7FOTYUZA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.197378931 (627h32m14.068461976s) [noise=true , address=77.90.31.14:4662 , piece_id=CU2OUXDQYV6ULLGUUUC464LIWQS45JTPB3CTMNQZC3JMIYH2SH4Q, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.260600915 (627h32m14.13168395s) [noise=true , address=77.90.31.10:4519 , piece_id=C25QYCH2VTZ6PE72D467NNLTBB74MU74XJQ36CQYTWO5TG22HG7A, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.292833670 (627h32m14.163916715s) [noise=true , address=77.90.31.19:4842 , piece_id=ZE2C5YWUKVXYLJTMKQRFYN6MVZ6EUOP4DLU4FEB6D6F3VXII7ZYA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.335927357 (627h32m14.207010392s) [noise=true , address=45.91.116.11:4119 , piece_id=LDYC6K6PPHDKOWGNEQXU464PFIBYSPWXMJZBEY7ELBRPP5YL57EA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.363700461 (627h32m14.234783496s) [noise=true , address=45.91.116.13:4156 , piece_id=LOJNZHUI2LDUHLQMKM2ZBRNRKVDEX7UXSBUR3WZJNOSCU36GVEUA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.375270873 (627h32m14.246353918s) [noise=true , address=77.90.31.21:4878 , piece_id=FX5WINCM4HKZVSKP6MTEKZOMOEWJRJ4J7E4HVGLG3CQSEDJXJLSQ, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.458059915 (627h32m14.32914295s) [noise=true , address=45.91.116.14:4189 , piece_id=4PUJ6YDAY364WVULRTHYBMG6MBLFDWKLLIPEVYFWK6HCQF3PC6LQ, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.577656003 (627h32m14.448739038s) [noise=true , address=77.90.31.18:4805 , piece_id=ZHDEUEKMLONGWMVC2FXD5OGIYUKO6QF5BSORJTSESP4JUJP7V2EQ, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.577676772 (627h32m14.448759807s) [seg_pos={0 0}, upload=549888]: Segment reached optimal threshold of 13 pieces.
Dec 31 19:01:14.578054592 (627h32m14.449137627s) [noise=true , address=45.91.116.12:4228 , piece_id=I5N63CF36AUCN2OGF66OV5GEVOQYQSUR3CS6YLJWWVMNGUL2W3XA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:47206->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:47206->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47206->45.91.116.12:4228: use of closed network connection
Dec 31 19:01:14.578062086 (627h32m14.449145121s) [noise=true , address=45.91.116.12:4228 , piece_id=I5N63CF36AUCN2OGF66OV5GEVOQYQSUR3CS6YLJWWVMNGUL2W3XA, seg_pos={0 0}, upload=549888]: Long tail cancellation due to optimal threshold
Dec 31 19:01:14.578143692 (627h32m14.449226747s) [noise=true , address=77.90.31.16:4736 , piece_id=PYWN35W6PGUC3NMTIROYVLH7UY2TJH3LRJWWEXBXSAXREX5QJX2Q, seg_pos={0 0}, upload=549888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF
Dec 31 19:01:14.578156606 (627h32m14.449239671s) [noise=true , address=77.90.31.16:4736 , piece_id=PYWN35W6PGUC3NMTIROYVLH7UY2TJH3LRJWWEXBXSAXREX5QJX2Q, seg_pos={0 0}, upload=549888]: Long tail cancellation due to optimal threshold
Dec 31 19:01:14.578161215 (627h32m14.44924425s) [noise=true , address=45.91.116.6:4974 , piece_id=R5FADLJDMRBYMO5AZL3MR6GUA55O4KTGJ6EWCB725JBHXSCMCYHA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:48032->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:48032->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48032->45.91.116.6:4974: use of closed network connection
Dec 31 19:01:14.578167326 (627h32m14.449250361s) [noise=true , address=45.91.116.6:4974 , piece_id=R5FADLJDMRBYMO5AZL3MR6GUA55O4KTGJ6EWCB725JBHXSCMCYHA, seg_pos={0 0}, upload=549888]: Long tail cancellation due to optimal threshold
Dec 31 19:01:14.578181263 (627h32m14.449264308s) [seg_pos={0 0}, upload=549888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 19:01:14.578277015 (627h32m14.44936005s) [upload=549888]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 19:01:14.578281534 (627h32m14.449364589s) [upload=549888]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 19:01:14.875461937 (627h32m14.746544982s) [upload=549888]: Done uploading segments...
Dec 31 19:01:14.875484871 (627h32m14.746567916s) [upload=549888]: Upload finished. err: <nil>
Duration: 5.795759587s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
5.893212545s trace:
https://jaeger.dev.storj.io/trace/9459b27b9ebbd
FAIL (2025-12-31 19:04:33 UTC)
orbiter warning: context deadline exceeded
Duration: 5.893240729s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [....X.......................................................] duration:
3.533117134s trace:
https://jaeger.dev.storj.io/trace/1c0a173ba9d72fc4
FAIL (2025-12-31 19:04:24 UTC)
orbiter warning: context deadline exceeded
Duration: 3.533209691s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [.........................X..................X.X.............] duration:
22.944705862s trace:
https://jaeger.dev.storj.io/trace/3fb4961d520cfd75
FAIL (2025-12-31 19:04:19 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12bYqoLXPqxPwKHkGRWx6RSpY8KSnS7QeKafAmtEpGJ5ThmJcDD, Piece ID: ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A) context deadline exceeded; piecestore: (Node ID: 1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, Piece ID: EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ) context deadline exceeded; piecestore: (Node ID: 1zF1SqsUdx5ikfqnG4SfeFukn8ZgX56sC7DGEPtPxZqEicws6o, Piece ID: PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA) context deadline exceeded
======LOGS======
, address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.958820533 (627h24m29.829903569s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.959307350 (627h24m29.830390386s) [noise=true , address=145.79.227.49:4349 , piece_id=R2IQZU7VSWQRVSJITAEMDNNJIPSJA3LBGC2X27AZ7GNZSN4XW2TA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.959608194 (627h24m29.830691229s) [noise=true , address=145.79.227.24:4588 , piece_id=5DP6OWR4UURMV22UA73ZIWUITQCEERVA6KE2K3T6TVDXAV2SO65A, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.959886383 (627h24m29.830969429s) [noise=true , address=64.49.13.26:4945 , piece_id=UMBXDB3DCEJERPVVINHZPPLJ7GKLDZ4PCZKBEOQD5RY5MB7LPCJQ, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.960144104 (627h24m29.831227139s) [noise=true , address=45.91.116.6:4008 , piece_id=64B6RIG5ANWZEVEMNU5HGC6X6E7EHBAIPDJN77UT6RQYYXRGVRWA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.960436030 (627h24m29.831519065s) [noise=true , address=145.79.227.47:4289 , piece_id=GMAPJV5LDECT7ZKYL23XWLDDTYHRAREVFZ2PEIH6OVJNTHUTTAHQ, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.960723447 (627h24m29.831806482s) [noise=true , address=145.79.227.42:4134 , piece_id=SWUYLD4EMMIYVLSWKSO3G3VOQ32ZGAI7NFX7KOURSIN3X6IZ4YJQ, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.960995856 (627h24m29.832078891s) [noise=true , address=145.79.227.40:4084 , piece_id=FD4PW56PCRS3QYHNXA6UQ3SXW2HNFIV3W53XFKMCQMAHDF542N3A, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.961374016 (627h24m29.832457051s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.961713552 (627h24m29.832796587s) [noise=true , address=64.49.13.137:4501 , piece_id=JBYIIB5EY4R2PJULBSLUFDQ3CMMYYIZHTS4BIX7GV5F2KY5OZC6A, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.962169091 (627h24m29.833252136s) [noise=true , address=145.79.227.2:4132 , piece_id=H2G6OZT52HEDBWWOSPU3VS5XIUT64XP5BIO2U7VAX5LNARERBQFA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.962533905 (627h24m29.83361695s) [noise=true , address=145.79.227.18:4027 , piece_id=RVLZBSJIG7UIDS6ITMGFQZJSJEKKDG4G6RFWYLXPUL7AEDZRZHMA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.963210673 (627h24m29.834293708s) [noise=true , address=145.79.227.35:4502 , piece_id=JDBTVMAGPR7M2TPMWJKMDQ562USDDYYPYFQMGEJSTNPARS6NAD5A, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.963755381 (627h24m29.834838426s) [noise=true , address=64.49.13.27:4957 , piece_id=ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.964023922 (627h24m29.835106957s) [noise=true , address=77.90.31.15:4730 , piece_id=PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.964592434 (627h24m29.835675469s) [noise=true , address=77.90.31.16:4753 , piece_id=EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.965227163 (627h24m29.836310208s) [noise=true , address=145.79.227.30:4770 , piece_id=TRAIDZYLZCU5UGHDYAJREYUELZQ5H5AFU4HHPYQFFKPXRZ5OJAQQ, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:31.410300275 (627h24m31.28138331s) [upload=549775]: Next returned nil segment
Dec 31 18:53:31.410315224 (627h24m31.281398259s) [upload=549775]: Waiting for error group managing segments...
Dec 31 18:53:31.788440578 (627h24m31.659523613s) [noise=true , address=64.49.13.26:4945 , piece_id=UMBXDB3DCEJERPVVINHZPPLJ7GKLDZ4PCZKBEOQD5RY5MB7LPCJQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.858179480 (627h24m31.729262526s) [noise=true , address=145.79.227.42:4134 , piece_id=SWUYLD4EMMIYVLSWKSO3G3VOQ32ZGAI7NFX7KOURSIN3X6IZ4YJQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.887866078 (627h24m31.758949113s) [noise=true , address=145.79.227.24:4588 , piece_id=5DP6OWR4UURMV22UA73ZIWUITQCEERVA6KE2K3T6TVDXAV2SO65A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.918774252 (627h24m31.789857297s) [noise=true , address=145.79.227.18:4027 , piece_id=RVLZBSJIG7UIDS6ITMGFQZJSJEKKDG4G6RFWYLXPUL7AEDZRZHMA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.922490298 (627h24m31.793573333s) [noise=true , address=145.79.227.35:4502 , piece_id=JDBTVMAGPR7M2TPMWJKMDQ562USDDYYPYFQMGEJSTNPARS6NAD5A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.924120612 (627h24m31.795203647s) [noise=true , address=145.79.227.2:4132 , piece_id=H2G6OZT52HEDBWWOSPU3VS5XIUT64XP5BIO2U7VAX5LNARERBQFA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.971450444 (627h24m31.842533479s) [noise=true , address=145.79.227.47:4289 , piece_id=GMAPJV5LDECT7ZKYL23XWLDDTYHRAREVFZ2PEIH6OVJNTHUTTAHQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.973634513 (627h24m31.844717548s) [noise=true , address=64.49.13.27:4957 , piece_id=ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.984147871 (627h24m31.855230906s) [noise=true , address=64.49.13.137:4501 , piece_id=JBYIIB5EY4R2PJULBSLUFDQ3CMMYYIZHTS4BIX7GV5F2KY5OZC6A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.002802704 (627h24m31.873885749s) [noise=true , address=77.90.31.15:4730 , piece_id=PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.113876665 (627h24m31.9849597s) [noise=true , address=77.90.31.16:4753 , piece_id=EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.307710471 (627h24m32.178793506s) [noise=true , address=45.91.116.6:4008 , piece_id=64B6RIG5ANWZEVEMNU5HGC6X6E7EHBAIPDJN77UT6RQYYXRGVRWA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.740289560 (627h24m32.611372605s) [noise=true , address=145.79.227.30:4770 , piece_id=TRAIDZYLZCU5UGHDYAJREYUELZQ5H5AFU4HHPYQFFKPXRZ5OJAQQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.744374660 (627h24m32.615457715s) [noise=true , address=145.79.227.49:4349 , piece_id=R2IQZU7VSWQRVSJITAEMDNNJIPSJA3LBGC2X27AZ7GNZSN4XW2TA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.744399677 (627h24m32.615482712s) [seg_pos={0 0}, upload=549775]: Segment reached optimal threshold of 14 pieces.
Dec 31 18:53:32.744624495 (627h24m32.61570753s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA): protocol: write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection; write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection
Dec 31 18:53:32.744633362 (627h24m32.615716397s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold
Dec 31 18:53:32.744938423 (627h24m32.616021468s) [noise=true , address=145.79.227.40:4084 , piece_id=FD4PW56PCRS3QYHNXA6UQ3SXW2HNFIV3W53XFKMCQMAHDF542N3A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.745025308 (627h24m32.616108343s) [noise=true , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gh7eXJMeRFyjBP2e9ZyTQVVE1mMoMFfSkQ9ssfxJyqG63C4gP): protocol: expected piece hash; context canceled; EOF
Dec 31 18:53:32.745035146 (627h24m32.616118181s) [noise=true , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold
Dec 31 18:53:32.745291255 (627h24m32.61637429s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu): protocol: expected piece hash; context canceled; EOF
Dec 31 18:53:32.745299210 (627h24m32.616382245s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold
Dec 31 18:53:32.745319939 (627h24m32.616402985s) [seg_pos={0 0}, upload=549775]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil>
Dec 31 18:53:32.745400913 (627h24m32.616483948s) [upload=549775]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 18:53:32.745406434 (627h24m32.616489479s) [upload=549775]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 18:53:33.044217872 (627h24m32.915300907s) [upload=549775]: Done uploading segments...
Dec 31 18:53:33.044236999 (627h24m32.915320034s) [upload=549775]: Upload finished. err: <nil>
Duration: 22.944758854s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [XX.X.X.X.X.X....XX.X....X...X.X...XXX..X..X........X...XX...] duration:
6.208293099s trace:
https://jaeger.dev.storj.io/trace/735677880864da8a
FAIL (2025-12-31 19:03:52 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: CU2OUXDQYV6ULLGUUUC464LIWQS45JTPB3CTMNQZC3JMIYH2SH4Q) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: 4PUJ6YDAY364WVULRTHYBMG6MBLFDWKLLIPEVYFWK6HCQF3PC6LQ) context deadline exceeded
======LOGS======
segments...
Dec 31 19:01:11.769976062 (627h32m11.641059107s) [upload=549888]: Got next segment. Inline: false
Dec 31 19:01:11.770023953 (627h32m11.641106998s) [upload=549888]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 19:01:11.770043050 (627h32m11.641126085s) [upload=549888]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 19:01:12.023231878 (627h32m11.894314913s) [seg_pos={0 0}, upload=549888]: Begin upload segment...
Dec 31 19:01:12.023296881 (627h32m11.894379916s) [seg_pos={0 0}, upload=549888]: Done begin upload segment.
Dec 31 19:01:12.023355753 (627h32m11.894438798s) [noise=true , address=77.90.31.10:4519 , piece_id=C25QYCH2VTZ6PE72D467NNLTBB74MU74XJQ36CQYTWO5TG22HG7A, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.023376493 (627h32m11.894459528s) [noise=true , address=77.90.31.14:4662 , piece_id=CU2OUXDQYV6ULLGUUUC464LIWQS45JTPB3CTMNQZC3JMIYH2SH4Q, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.023952419 (627h32m11.895035454s) [noise=true , address=45.91.116.11:4119 , piece_id=LDYC6K6PPHDKOWGNEQXU464PFIBYSPWXMJZBEY7ELBRPP5YL57EA, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.024015850 (627h32m11.895098885s) [noise=true , address=77.90.31.16:4736 , piece_id=PYWN35W6PGUC3NMTIROYVLH7UY2TJH3LRJWWEXBXSAXREX5QJX2Q, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.024311754 (627h32m11.895394789s) [noise=true , address=145.79.227.46:4603 , piece_id=RTKSKQ4L63I7MBTLOAMZE36DNSPQ2ZPXV64KRMQPNWCHHODZIYGA, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.024431361 (627h32m11.895514397s) [noise=true , address=145.79.227.6:4259 , piece_id=OQAYR6PIVUKBG2A3OKZBZX3H7ZJFEUR256I3IU6QWYQ4X7PVZGSQ, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.024577730 (627h32m11.895660765s) [noise=true , address=145.79.227.14:4840 , piece_id=QYG4BKZ54RFDZGVXEYNGXRAL4INUYCE4CLPWKCETGL7DSBOKEFKA, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.024765618 (627h32m11.895848653s) [noise=true , address=45.91.116.14:4189 , piece_id=4PUJ6YDAY364WVULRTHYBMG6MBLFDWKLLIPEVYFWK6HCQF3PC6LQ, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.024809993 (627h32m11.895893028s) [noise=true , address=145.79.227.22:4509 , piece_id=WVFB7RFZSVW2M446KJGVIJ4FUGYC7QLEAB6TYY6J2O563KXBFT6A, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.025153747 (627h32m11.896236782s) [noise=true , address=77.90.31.19:4842 , piece_id=ZE2C5YWUKVXYLJTMKQRFYN6MVZ6EUOP4DLU4FEB6D6F3VXII7ZYA, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.025220134 (627h32m11.896303179s) [noise=true , address=45.91.116.6:4974 , piece_id=R5FADLJDMRBYMO5AZL3MR6GUA55O4KTGJ6EWCB725JBHXSCMCYHA, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.025492613 (627h32m11.896575648s) [noise=true , address=77.90.31.18:4805 , piece_id=ZHDEUEKMLONGWMVC2FXD5OGIYUKO6QF5BSORJTSESP4JUJP7V2EQ, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.025786592 (627h32m11.896869627s) [noise=true , address=45.91.116.13:4156 , piece_id=LOJNZHUI2LDUHLQMKM2ZBRNRKVDEX7UXSBUR3WZJNOSCU36GVEUA, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.026165153 (627h32m11.897248198s) [noise=true , address=145.79.227.48:4303 , piece_id=HGCHG7IT4ZBWBYCPULLYDC6M2CAOCNR4PSIH4JU3Y5WP7FOTYUZA, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.026497455 (627h32m11.8975805s) [noise=true , address=45.91.116.12:4228 , piece_id=I5N63CF36AUCN2OGF66OV5GEVOQYQSUR3CS6YLJWWVMNGUL2W3XA, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.026680595 (627h32m11.89776364s) [noise=true , address=77.90.31.21:4878 , piece_id=FX5WINCM4HKZVSKP6MTEKZOMOEWJRJ4J7E4HVGLG3CQSEDJXJLSQ, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:13.730039170 (627h32m13.601122215s) [upload=549888]: Next returned nil segment
Dec 31 19:01:13.730056022 (627h32m13.601139057s) [upload=549888]: Waiting for error group managing segments...
Dec 31 19:01:14.011727154 (627h32m13.882810199s) [noise=true , address=145.79.227.46:4603 , piece_id=RTKSKQ4L63I7MBTLOAMZE36DNSPQ2ZPXV64KRMQPNWCHHODZIYGA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.015213612 (627h32m13.886296647s) [noise=true , address=145.79.227.22:4509 , piece_id=WVFB7RFZSVW2M446KJGVIJ4FUGYC7QLEAB6TYY6J2O563KXBFT6A, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.016168130 (627h32m13.887251165s) [noise=true , address=145.79.227.6:4259 , piece_id=OQAYR6PIVUKBG2A3OKZBZX3H7ZJFEUR256I3IU6QWYQ4X7PVZGSQ, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.067093519 (627h32m13.938176554s) [noise=true , address=145.79.227.14:4840 , piece_id=QYG4BKZ54RFDZGVXEYNGXRAL4INUYCE4CLPWKCETGL7DSBOKEFKA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.094476500 (627h32m13.965559545s) [noise=true , address=145.79.227.48:4303 , piece_id=HGCHG7IT4ZBWBYCPULLYDC6M2CAOCNR4PSIH4JU3Y5WP7FOTYUZA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.197378931 (627h32m14.068461976s) [noise=true , address=77.90.31.14:4662 , piece_id=CU2OUXDQYV6ULLGUUUC464LIWQS45JTPB3CTMNQZC3JMIYH2SH4Q, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.260600915 (627h32m14.13168395s) [noise=true , address=77.90.31.10:4519 , piece_id=C25QYCH2VTZ6PE72D467NNLTBB74MU74XJQ36CQYTWO5TG22HG7A, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.292833670 (627h32m14.163916715s) [noise=true , address=77.90.31.19:4842 , piece_id=ZE2C5YWUKVXYLJTMKQRFYN6MVZ6EUOP4DLU4FEB6D6F3VXII7ZYA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.335927357 (627h32m14.207010392s) [noise=true , address=45.91.116.11:4119 , piece_id=LDYC6K6PPHDKOWGNEQXU464PFIBYSPWXMJZBEY7ELBRPP5YL57EA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.363700461 (627h32m14.234783496s) [noise=true , address=45.91.116.13:4156 , piece_id=LOJNZHUI2LDUHLQMKM2ZBRNRKVDEX7UXSBUR3WZJNOSCU36GVEUA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.375270873 (627h32m14.246353918s) [noise=true , address=77.90.31.21:4878 , piece_id=FX5WINCM4HKZVSKP6MTEKZOMOEWJRJ4J7E4HVGLG3CQSEDJXJLSQ, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.458059915 (627h32m14.32914295s) [noise=true , address=45.91.116.14:4189 , piece_id=4PUJ6YDAY364WVULRTHYBMG6MBLFDWKLLIPEVYFWK6HCQF3PC6LQ, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.577656003 (627h32m14.448739038s) [noise=true , address=77.90.31.18:4805 , piece_id=ZHDEUEKMLONGWMVC2FXD5OGIYUKO6QF5BSORJTSESP4JUJP7V2EQ, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.577676772 (627h32m14.448759807s) [seg_pos={0 0}, upload=549888]: Segment reached optimal threshold of 13 pieces.
Dec 31 19:01:14.578054592 (627h32m14.449137627s) [noise=true , address=45.91.116.12:4228 , piece_id=I5N63CF36AUCN2OGF66OV5GEVOQYQSUR3CS6YLJWWVMNGUL2W3XA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:47206->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:47206->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47206->45.91.116.12:4228: use of closed network connection
Dec 31 19:01:14.578062086 (627h32m14.449145121s) [noise=true , address=45.91.116.12:4228 , piece_id=I5N63CF36AUCN2OGF66OV5GEVOQYQSUR3CS6YLJWWVMNGUL2W3XA, seg_pos={0 0}, upload=549888]: Long tail cancellation due to optimal threshold
Dec 31 19:01:14.578143692 (627h32m14.449226747s) [noise=true , address=77.90.31.16:4736 , piece_id=PYWN35W6PGUC3NMTIROYVLH7UY2TJH3LRJWWEXBXSAXREX5QJX2Q, seg_pos={0 0}, upload=549888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF
Dec 31 19:01:14.578156606 (627h32m14.449239671s) [noise=true , address=77.90.31.16:4736 , piece_id=PYWN35W6PGUC3NMTIROYVLH7UY2TJH3LRJWWEXBXSAXREX5QJX2Q, seg_pos={0 0}, upload=549888]: Long tail cancellation due to optimal threshold
Dec 31 19:01:14.578161215 (627h32m14.44924425s) [noise=true , address=45.91.116.6:4974 , piece_id=R5FADLJDMRBYMO5AZL3MR6GUA55O4KTGJ6EWCB725JBHXSCMCYHA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:48032->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:48032->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48032->45.91.116.6:4974: use of closed network connection
Dec 31 19:01:14.578167326 (627h32m14.449250361s) [noise=true , address=45.91.116.6:4974 , piece_id=R5FADLJDMRBYMO5AZL3MR6GUA55O4KTGJ6EWCB725JBHXSCMCYHA, seg_pos={0 0}, upload=549888]: Long tail cancellation due to optimal threshold
Dec 31 19:01:14.578181263 (627h32m14.449264308s) [seg_pos={0 0}, upload=549888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 19:01:14.578277015 (627h32m14.44936005s) [upload=549888]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 19:01:14.578281534 (627h32m14.449364589s) [upload=549888]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 19:01:14.875461937 (627h32m14.746544982s) [upload=549888]: Done uploading segments...
Dec 31 19:01:14.875484871 (627h32m14.746567916s) [upload=549888]: Upload finished. err: <nil>
Duration: 6.208310633s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [..........................X..................X.X............] duration:
28.406865433s trace:
https://jaeger.dev.storj.io/trace/3fb4961d520cfd75
FAIL (2025-12-31 19:03:42 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12bYqoLXPqxPwKHkGRWx6RSpY8KSnS7QeKafAmtEpGJ5ThmJcDD, Piece ID: ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A) context deadline exceeded; piecestore: (Node ID: 1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, Piece ID: EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ) context deadline exceeded; piecestore: (Node ID: 1zF1SqsUdx5ikfqnG4SfeFukn8ZgX56sC7DGEPtPxZqEicws6o, Piece ID: PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA) context deadline exceeded
======LOGS======
, address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.958820533 (627h24m29.829903569s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.959307350 (627h24m29.830390386s) [noise=true , address=145.79.227.49:4349 , piece_id=R2IQZU7VSWQRVSJITAEMDNNJIPSJA3LBGC2X27AZ7GNZSN4XW2TA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.959608194 (627h24m29.830691229s) [noise=true , address=145.79.227.24:4588 , piece_id=5DP6OWR4UURMV22UA73ZIWUITQCEERVA6KE2K3T6TVDXAV2SO65A, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.959886383 (627h24m29.830969429s) [noise=true , address=64.49.13.26:4945 , piece_id=UMBXDB3DCEJERPVVINHZPPLJ7GKLDZ4PCZKBEOQD5RY5MB7LPCJQ, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.960144104 (627h24m29.831227139s) [noise=true , address=45.91.116.6:4008 , piece_id=64B6RIG5ANWZEVEMNU5HGC6X6E7EHBAIPDJN77UT6RQYYXRGVRWA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.960436030 (627h24m29.831519065s) [noise=true , address=145.79.227.47:4289 , piece_id=GMAPJV5LDECT7ZKYL23XWLDDTYHRAREVFZ2PEIH6OVJNTHUTTAHQ, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.960723447 (627h24m29.831806482s) [noise=true , address=145.79.227.42:4134 , piece_id=SWUYLD4EMMIYVLSWKSO3G3VOQ32ZGAI7NFX7KOURSIN3X6IZ4YJQ, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.960995856 (627h24m29.832078891s) [noise=true , address=145.79.227.40:4084 , piece_id=FD4PW56PCRS3QYHNXA6UQ3SXW2HNFIV3W53XFKMCQMAHDF542N3A, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.961374016 (627h24m29.832457051s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.961713552 (627h24m29.832796587s) [noise=true , address=64.49.13.137:4501 , piece_id=JBYIIB5EY4R2PJULBSLUFDQ3CMMYYIZHTS4BIX7GV5F2KY5OZC6A, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.962169091 (627h24m29.833252136s) [noise=true , address=145.79.227.2:4132 , piece_id=H2G6OZT52HEDBWWOSPU3VS5XIUT64XP5BIO2U7VAX5LNARERBQFA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.962533905 (627h24m29.83361695s) [noise=true , address=145.79.227.18:4027 , piece_id=RVLZBSJIG7UIDS6ITMGFQZJSJEKKDG4G6RFWYLXPUL7AEDZRZHMA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.963210673 (627h24m29.834293708s) [noise=true , address=145.79.227.35:4502 , piece_id=JDBTVMAGPR7M2TPMWJKMDQ562USDDYYPYFQMGEJSTNPARS6NAD5A, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.963755381 (627h24m29.834838426s) [noise=true , address=64.49.13.27:4957 , piece_id=ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.964023922 (627h24m29.835106957s) [noise=true , address=77.90.31.15:4730 , piece_id=PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.964592434 (627h24m29.835675469s) [noise=true , address=77.90.31.16:4753 , piece_id=EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.965227163 (627h24m29.836310208s) [noise=true , address=145.79.227.30:4770 , piece_id=TRAIDZYLZCU5UGHDYAJREYUELZQ5H5AFU4HHPYQFFKPXRZ5OJAQQ, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:31.410300275 (627h24m31.28138331s) [upload=549775]: Next returned nil segment
Dec 31 18:53:31.410315224 (627h24m31.281398259s) [upload=549775]: Waiting for error group managing segments...
Dec 31 18:53:31.788440578 (627h24m31.659523613s) [noise=true , address=64.49.13.26:4945 , piece_id=UMBXDB3DCEJERPVVINHZPPLJ7GKLDZ4PCZKBEOQD5RY5MB7LPCJQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.858179480 (627h24m31.729262526s) [noise=true , address=145.79.227.42:4134 , piece_id=SWUYLD4EMMIYVLSWKSO3G3VOQ32ZGAI7NFX7KOURSIN3X6IZ4YJQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.887866078 (627h24m31.758949113s) [noise=true , address=145.79.227.24:4588 , piece_id=5DP6OWR4UURMV22UA73ZIWUITQCEERVA6KE2K3T6TVDXAV2SO65A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.918774252 (627h24m31.789857297s) [noise=true , address=145.79.227.18:4027 , piece_id=RVLZBSJIG7UIDS6ITMGFQZJSJEKKDG4G6RFWYLXPUL7AEDZRZHMA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.922490298 (627h24m31.793573333s) [noise=true , address=145.79.227.35:4502 , piece_id=JDBTVMAGPR7M2TPMWJKMDQ562USDDYYPYFQMGEJSTNPARS6NAD5A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.924120612 (627h24m31.795203647s) [noise=true , address=145.79.227.2:4132 , piece_id=H2G6OZT52HEDBWWOSPU3VS5XIUT64XP5BIO2U7VAX5LNARERBQFA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.971450444 (627h24m31.842533479s) [noise=true , address=145.79.227.47:4289 , piece_id=GMAPJV5LDECT7ZKYL23XWLDDTYHRAREVFZ2PEIH6OVJNTHUTTAHQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.973634513 (627h24m31.844717548s) [noise=true , address=64.49.13.27:4957 , piece_id=ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.984147871 (627h24m31.855230906s) [noise=true , address=64.49.13.137:4501 , piece_id=JBYIIB5EY4R2PJULBSLUFDQ3CMMYYIZHTS4BIX7GV5F2KY5OZC6A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.002802704 (627h24m31.873885749s) [noise=true , address=77.90.31.15:4730 , piece_id=PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.113876665 (627h24m31.9849597s) [noise=true , address=77.90.31.16:4753 , piece_id=EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.307710471 (627h24m32.178793506s) [noise=true , address=45.91.116.6:4008 , piece_id=64B6RIG5ANWZEVEMNU5HGC6X6E7EHBAIPDJN77UT6RQYYXRGVRWA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.740289560 (627h24m32.611372605s) [noise=true , address=145.79.227.30:4770 , piece_id=TRAIDZYLZCU5UGHDYAJREYUELZQ5H5AFU4HHPYQFFKPXRZ5OJAQQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.744374660 (627h24m32.615457715s) [noise=true , address=145.79.227.49:4349 , piece_id=R2IQZU7VSWQRVSJITAEMDNNJIPSJA3LBGC2X27AZ7GNZSN4XW2TA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.744399677 (627h24m32.615482712s) [seg_pos={0 0}, upload=549775]: Segment reached optimal threshold of 14 pieces.
Dec 31 18:53:32.744624495 (627h24m32.61570753s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA): protocol: write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection; write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection
Dec 31 18:53:32.744633362 (627h24m32.615716397s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold
Dec 31 18:53:32.744938423 (627h24m32.616021468s) [noise=true , address=145.79.227.40:4084 , piece_id=FD4PW56PCRS3QYHNXA6UQ3SXW2HNFIV3W53XFKMCQMAHDF542N3A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.745025308 (627h24m32.616108343s) [noise=true , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gh7eXJMeRFyjBP2e9ZyTQVVE1mMoMFfSkQ9ssfxJyqG63C4gP): protocol: expected piece hash; context canceled; EOF
Dec 31 18:53:32.745035146 (627h24m32.616118181s) [noise=true , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold
Dec 31 18:53:32.745291255 (627h24m32.61637429s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu): protocol: expected piece hash; context canceled; EOF
Dec 31 18:53:32.745299210 (627h24m32.616382245s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold
Dec 31 18:53:32.745319939 (627h24m32.616402985s) [seg_pos={0 0}, upload=549775]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil>
Dec 31 18:53:32.745400913 (627h24m32.616483948s) [upload=549775]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 18:53:32.745406434 (627h24m32.616489479s) [upload=549775]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 18:53:33.044217872 (627h24m32.915300907s) [upload=549775]: Done uploading segments...
Dec 31 18:53:33.044236999 (627h24m32.915320034s) [upload=549775]: Upload finished. err: <nil>
Duration: 28.406881374s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
5.905114304s trace:
https://jaeger.dev.storj.io/trace/9459b27b9ebbd
FAIL (2025-12-31 19:03:39 UTC)
orbiter warning: context deadline exceeded
Duration: 5.905141155s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.....X......................................................] duration:
3.549761148s trace:
https://jaeger.dev.storj.io/trace/1c0a173ba9d72fc4
FAIL (2025-12-31 19:03:26 UTC)
orbiter warning: context deadline exceeded
Duration: 3.549801886s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
6.311524344s trace:
https://jaeger.dev.storj.io/trace/9459b27b9ebbd
FAIL (2025-12-31 19:03:15 UTC)
orbiter warning: context deadline exceeded
Duration: 6.311555784s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [...........................X..................X.X...........] duration:
47.921248069s trace:
https://jaeger.dev.storj.io/trace/3fb4961d520cfd75
FAIL (2025-12-31 19:03:13 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12bYqoLXPqxPwKHkGRWx6RSpY8KSnS7QeKafAmtEpGJ5ThmJcDD, Piece ID: ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A) context deadline exceeded; piecestore: (Node ID: 1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, Piece ID: EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ) context deadline exceeded; piecestore: (Node ID: 1zF1SqsUdx5ikfqnG4SfeFukn8ZgX56sC7DGEPtPxZqEicws6o, Piece ID: PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA) context deadline exceeded
======LOGS======
, address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.958820533 (627h24m29.829903569s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.959307350 (627h24m29.830390386s) [noise=true , address=145.79.227.49:4349 , piece_id=R2IQZU7VSWQRVSJITAEMDNNJIPSJA3LBGC2X27AZ7GNZSN4XW2TA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.959608194 (627h24m29.830691229s) [noise=true , address=145.79.227.24:4588 , piece_id=5DP6OWR4UURMV22UA73ZIWUITQCEERVA6KE2K3T6TVDXAV2SO65A, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.959886383 (627h24m29.830969429s) [noise=true , address=64.49.13.26:4945 , piece_id=UMBXDB3DCEJERPVVINHZPPLJ7GKLDZ4PCZKBEOQD5RY5MB7LPCJQ, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.960144104 (627h24m29.831227139s) [noise=true , address=45.91.116.6:4008 , piece_id=64B6RIG5ANWZEVEMNU5HGC6X6E7EHBAIPDJN77UT6RQYYXRGVRWA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.960436030 (627h24m29.831519065s) [noise=true , address=145.79.227.47:4289 , piece_id=GMAPJV5LDECT7ZKYL23XWLDDTYHRAREVFZ2PEIH6OVJNTHUTTAHQ, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.960723447 (627h24m29.831806482s) [noise=true , address=145.79.227.42:4134 , piece_id=SWUYLD4EMMIYVLSWKSO3G3VOQ32ZGAI7NFX7KOURSIN3X6IZ4YJQ, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.960995856 (627h24m29.832078891s) [noise=true , address=145.79.227.40:4084 , piece_id=FD4PW56PCRS3QYHNXA6UQ3SXW2HNFIV3W53XFKMCQMAHDF542N3A, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.961374016 (627h24m29.832457051s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.961713552 (627h24m29.832796587s) [noise=true , address=64.49.13.137:4501 , piece_id=JBYIIB5EY4R2PJULBSLUFDQ3CMMYYIZHTS4BIX7GV5F2KY5OZC6A, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.962169091 (627h24m29.833252136s) [noise=true , address=145.79.227.2:4132 , piece_id=H2G6OZT52HEDBWWOSPU3VS5XIUT64XP5BIO2U7VAX5LNARERBQFA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.962533905 (627h24m29.83361695s) [noise=true , address=145.79.227.18:4027 , piece_id=RVLZBSJIG7UIDS6ITMGFQZJSJEKKDG4G6RFWYLXPUL7AEDZRZHMA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.963210673 (627h24m29.834293708s) [noise=true , address=145.79.227.35:4502 , piece_id=JDBTVMAGPR7M2TPMWJKMDQ562USDDYYPYFQMGEJSTNPARS6NAD5A, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.963755381 (627h24m29.834838426s) [noise=true , address=64.49.13.27:4957 , piece_id=ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.964023922 (627h24m29.835106957s) [noise=true , address=77.90.31.15:4730 , piece_id=PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.964592434 (627h24m29.835675469s) [noise=true , address=77.90.31.16:4753 , piece_id=EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.965227163 (627h24m29.836310208s) [noise=true , address=145.79.227.30:4770 , piece_id=TRAIDZYLZCU5UGHDYAJREYUELZQ5H5AFU4HHPYQFFKPXRZ5OJAQQ, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:31.410300275 (627h24m31.28138331s) [upload=549775]: Next returned nil segment
Dec 31 18:53:31.410315224 (627h24m31.281398259s) [upload=549775]: Waiting for error group managing segments...
Dec 31 18:53:31.788440578 (627h24m31.659523613s) [noise=true , address=64.49.13.26:4945 , piece_id=UMBXDB3DCEJERPVVINHZPPLJ7GKLDZ4PCZKBEOQD5RY5MB7LPCJQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.858179480 (627h24m31.729262526s) [noise=true , address=145.79.227.42:4134 , piece_id=SWUYLD4EMMIYVLSWKSO3G3VOQ32ZGAI7NFX7KOURSIN3X6IZ4YJQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.887866078 (627h24m31.758949113s) [noise=true , address=145.79.227.24:4588 , piece_id=5DP6OWR4UURMV22UA73ZIWUITQCEERVA6KE2K3T6TVDXAV2SO65A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.918774252 (627h24m31.789857297s) [noise=true , address=145.79.227.18:4027 , piece_id=RVLZBSJIG7UIDS6ITMGFQZJSJEKKDG4G6RFWYLXPUL7AEDZRZHMA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.922490298 (627h24m31.793573333s) [noise=true , address=145.79.227.35:4502 , piece_id=JDBTVMAGPR7M2TPMWJKMDQ562USDDYYPYFQMGEJSTNPARS6NAD5A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.924120612 (627h24m31.795203647s) [noise=true , address=145.79.227.2:4132 , piece_id=H2G6OZT52HEDBWWOSPU3VS5XIUT64XP5BIO2U7VAX5LNARERBQFA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.971450444 (627h24m31.842533479s) [noise=true , address=145.79.227.47:4289 , piece_id=GMAPJV5LDECT7ZKYL23XWLDDTYHRAREVFZ2PEIH6OVJNTHUTTAHQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.973634513 (627h24m31.844717548s) [noise=true , address=64.49.13.27:4957 , piece_id=ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.984147871 (627h24m31.855230906s) [noise=true , address=64.49.13.137:4501 , piece_id=JBYIIB5EY4R2PJULBSLUFDQ3CMMYYIZHTS4BIX7GV5F2KY5OZC6A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.002802704 (627h24m31.873885749s) [noise=true , address=77.90.31.15:4730 , piece_id=PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.113876665 (627h24m31.9849597s) [noise=true , address=77.90.31.16:4753 , piece_id=EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.307710471 (627h24m32.178793506s) [noise=true , address=45.91.116.6:4008 , piece_id=64B6RIG5ANWZEVEMNU5HGC6X6E7EHBAIPDJN77UT6RQYYXRGVRWA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.740289560 (627h24m32.611372605s) [noise=true , address=145.79.227.30:4770 , piece_id=TRAIDZYLZCU5UGHDYAJREYUELZQ5H5AFU4HHPYQFFKPXRZ5OJAQQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.744374660 (627h24m32.615457715s) [noise=true , address=145.79.227.49:4349 , piece_id=R2IQZU7VSWQRVSJITAEMDNNJIPSJA3LBGC2X27AZ7GNZSN4XW2TA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.744399677 (627h24m32.615482712s) [seg_pos={0 0}, upload=549775]: Segment reached optimal threshold of 14 pieces.
Dec 31 18:53:32.744624495 (627h24m32.61570753s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA): protocol: write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection; write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection
Dec 31 18:53:32.744633362 (627h24m32.615716397s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold
Dec 31 18:53:32.744938423 (627h24m32.616021468s) [noise=true , address=145.79.227.40:4084 , piece_id=FD4PW56PCRS3QYHNXA6UQ3SXW2HNFIV3W53XFKMCQMAHDF542N3A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.745025308 (627h24m32.616108343s) [noise=true , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gh7eXJMeRFyjBP2e9ZyTQVVE1mMoMFfSkQ9ssfxJyqG63C4gP): protocol: expected piece hash; context canceled; EOF
Dec 31 18:53:32.745035146 (627h24m32.616118181s) [noise=true , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold
Dec 31 18:53:32.745291255 (627h24m32.61637429s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu): protocol: expected piece hash; context canceled; EOF
Dec 31 18:53:32.745299210 (627h24m32.616382245s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold
Dec 31 18:53:32.745319939 (627h24m32.616402985s) [seg_pos={0 0}, upload=549775]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil>
Dec 31 18:53:32.745400913 (627h24m32.616483948s) [upload=549775]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 18:53:32.745406434 (627h24m32.616489479s) [upload=549775]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 18:53:33.044217872 (627h24m32.915300907s) [upload=549775]: Done uploading segments...
Dec 31 18:53:33.044236999 (627h24m32.915320034s) [upload=549775]: Upload finished. err: <nil>
Duration: 47.921280771s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
22 out of 60 is failed (threshold=3) [XXX.X.X.X.X.X....XX.X....X...X.X...XXX..X..X........X...XX..] duration:
8.815822909s trace:
https://jaeger.dev.storj.io/trace/735677880864da8a
FAIL (2025-12-31 19:02:53 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: CU2OUXDQYV6ULLGUUUC464LIWQS45JTPB3CTMNQZC3JMIYH2SH4Q) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: 4PUJ6YDAY364WVULRTHYBMG6MBLFDWKLLIPEVYFWK6HCQF3PC6LQ) context deadline exceeded
======LOGS======
segments...
Dec 31 19:01:11.769976062 (627h32m11.641059107s) [upload=549888]: Got next segment. Inline: false
Dec 31 19:01:11.770023953 (627h32m11.641106998s) [upload=549888]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 19:01:11.770043050 (627h32m11.641126085s) [upload=549888]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 19:01:12.023231878 (627h32m11.894314913s) [seg_pos={0 0}, upload=549888]: Begin upload segment...
Dec 31 19:01:12.023296881 (627h32m11.894379916s) [seg_pos={0 0}, upload=549888]: Done begin upload segment.
Dec 31 19:01:12.023355753 (627h32m11.894438798s) [noise=true , address=77.90.31.10:4519 , piece_id=C25QYCH2VTZ6PE72D467NNLTBB74MU74XJQ36CQYTWO5TG22HG7A, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.023376493 (627h32m11.894459528s) [noise=true , address=77.90.31.14:4662 , piece_id=CU2OUXDQYV6ULLGUUUC464LIWQS45JTPB3CTMNQZC3JMIYH2SH4Q, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.023952419 (627h32m11.895035454s) [noise=true , address=45.91.116.11:4119 , piece_id=LDYC6K6PPHDKOWGNEQXU464PFIBYSPWXMJZBEY7ELBRPP5YL57EA, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.024015850 (627h32m11.895098885s) [noise=true , address=77.90.31.16:4736 , piece_id=PYWN35W6PGUC3NMTIROYVLH7UY2TJH3LRJWWEXBXSAXREX5QJX2Q, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.024311754 (627h32m11.895394789s) [noise=true , address=145.79.227.46:4603 , piece_id=RTKSKQ4L63I7MBTLOAMZE36DNSPQ2ZPXV64KRMQPNWCHHODZIYGA, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.024431361 (627h32m11.895514397s) [noise=true , address=145.79.227.6:4259 , piece_id=OQAYR6PIVUKBG2A3OKZBZX3H7ZJFEUR256I3IU6QWYQ4X7PVZGSQ, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.024577730 (627h32m11.895660765s) [noise=true , address=145.79.227.14:4840 , piece_id=QYG4BKZ54RFDZGVXEYNGXRAL4INUYCE4CLPWKCETGL7DSBOKEFKA, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.024765618 (627h32m11.895848653s) [noise=true , address=45.91.116.14:4189 , piece_id=4PUJ6YDAY364WVULRTHYBMG6MBLFDWKLLIPEVYFWK6HCQF3PC6LQ, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.024809993 (627h32m11.895893028s) [noise=true , address=145.79.227.22:4509 , piece_id=WVFB7RFZSVW2M446KJGVIJ4FUGYC7QLEAB6TYY6J2O563KXBFT6A, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.025153747 (627h32m11.896236782s) [noise=true , address=77.90.31.19:4842 , piece_id=ZE2C5YWUKVXYLJTMKQRFYN6MVZ6EUOP4DLU4FEB6D6F3VXII7ZYA, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.025220134 (627h32m11.896303179s) [noise=true , address=45.91.116.6:4974 , piece_id=R5FADLJDMRBYMO5AZL3MR6GUA55O4KTGJ6EWCB725JBHXSCMCYHA, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.025492613 (627h32m11.896575648s) [noise=true , address=77.90.31.18:4805 , piece_id=ZHDEUEKMLONGWMVC2FXD5OGIYUKO6QF5BSORJTSESP4JUJP7V2EQ, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.025786592 (627h32m11.896869627s) [noise=true , address=45.91.116.13:4156 , piece_id=LOJNZHUI2LDUHLQMKM2ZBRNRKVDEX7UXSBUR3WZJNOSCU36GVEUA, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.026165153 (627h32m11.897248198s) [noise=true , address=145.79.227.48:4303 , piece_id=HGCHG7IT4ZBWBYCPULLYDC6M2CAOCNR4PSIH4JU3Y5WP7FOTYUZA, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.026497455 (627h32m11.8975805s) [noise=true , address=45.91.116.12:4228 , piece_id=I5N63CF36AUCN2OGF66OV5GEVOQYQSUR3CS6YLJWWVMNGUL2W3XA, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.026680595 (627h32m11.89776364s) [noise=true , address=77.90.31.21:4878 , piece_id=FX5WINCM4HKZVSKP6MTEKZOMOEWJRJ4J7E4HVGLG3CQSEDJXJLSQ, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:13.730039170 (627h32m13.601122215s) [upload=549888]: Next returned nil segment
Dec 31 19:01:13.730056022 (627h32m13.601139057s) [upload=549888]: Waiting for error group managing segments...
Dec 31 19:01:14.011727154 (627h32m13.882810199s) [noise=true , address=145.79.227.46:4603 , piece_id=RTKSKQ4L63I7MBTLOAMZE36DNSPQ2ZPXV64KRMQPNWCHHODZIYGA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.015213612 (627h32m13.886296647s) [noise=true , address=145.79.227.22:4509 , piece_id=WVFB7RFZSVW2M446KJGVIJ4FUGYC7QLEAB6TYY6J2O563KXBFT6A, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.016168130 (627h32m13.887251165s) [noise=true , address=145.79.227.6:4259 , piece_id=OQAYR6PIVUKBG2A3OKZBZX3H7ZJFEUR256I3IU6QWYQ4X7PVZGSQ, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.067093519 (627h32m13.938176554s) [noise=true , address=145.79.227.14:4840 , piece_id=QYG4BKZ54RFDZGVXEYNGXRAL4INUYCE4CLPWKCETGL7DSBOKEFKA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.094476500 (627h32m13.965559545s) [noise=true , address=145.79.227.48:4303 , piece_id=HGCHG7IT4ZBWBYCPULLYDC6M2CAOCNR4PSIH4JU3Y5WP7FOTYUZA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.197378931 (627h32m14.068461976s) [noise=true , address=77.90.31.14:4662 , piece_id=CU2OUXDQYV6ULLGUUUC464LIWQS45JTPB3CTMNQZC3JMIYH2SH4Q, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.260600915 (627h32m14.13168395s) [noise=true , address=77.90.31.10:4519 , piece_id=C25QYCH2VTZ6PE72D467NNLTBB74MU74XJQ36CQYTWO5TG22HG7A, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.292833670 (627h32m14.163916715s) [noise=true , address=77.90.31.19:4842 , piece_id=ZE2C5YWUKVXYLJTMKQRFYN6MVZ6EUOP4DLU4FEB6D6F3VXII7ZYA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.335927357 (627h32m14.207010392s) [noise=true , address=45.91.116.11:4119 , piece_id=LDYC6K6PPHDKOWGNEQXU464PFIBYSPWXMJZBEY7ELBRPP5YL57EA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.363700461 (627h32m14.234783496s) [noise=true , address=45.91.116.13:4156 , piece_id=LOJNZHUI2LDUHLQMKM2ZBRNRKVDEX7UXSBUR3WZJNOSCU36GVEUA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.375270873 (627h32m14.246353918s) [noise=true , address=77.90.31.21:4878 , piece_id=FX5WINCM4HKZVSKP6MTEKZOMOEWJRJ4J7E4HVGLG3CQSEDJXJLSQ, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.458059915 (627h32m14.32914295s) [noise=true , address=45.91.116.14:4189 , piece_id=4PUJ6YDAY364WVULRTHYBMG6MBLFDWKLLIPEVYFWK6HCQF3PC6LQ, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.577656003 (627h32m14.448739038s) [noise=true , address=77.90.31.18:4805 , piece_id=ZHDEUEKMLONGWMVC2FXD5OGIYUKO6QF5BSORJTSESP4JUJP7V2EQ, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.577676772 (627h32m14.448759807s) [seg_pos={0 0}, upload=549888]: Segment reached optimal threshold of 13 pieces.
Dec 31 19:01:14.578054592 (627h32m14.449137627s) [noise=true , address=45.91.116.12:4228 , piece_id=I5N63CF36AUCN2OGF66OV5GEVOQYQSUR3CS6YLJWWVMNGUL2W3XA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:47206->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:47206->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47206->45.91.116.12:4228: use of closed network connection
Dec 31 19:01:14.578062086 (627h32m14.449145121s) [noise=true , address=45.91.116.12:4228 , piece_id=I5N63CF36AUCN2OGF66OV5GEVOQYQSUR3CS6YLJWWVMNGUL2W3XA, seg_pos={0 0}, upload=549888]: Long tail cancellation due to optimal threshold
Dec 31 19:01:14.578143692 (627h32m14.449226747s) [noise=true , address=77.90.31.16:4736 , piece_id=PYWN35W6PGUC3NMTIROYVLH7UY2TJH3LRJWWEXBXSAXREX5QJX2Q, seg_pos={0 0}, upload=549888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF
Dec 31 19:01:14.578156606 (627h32m14.449239671s) [noise=true , address=77.90.31.16:4736 , piece_id=PYWN35W6PGUC3NMTIROYVLH7UY2TJH3LRJWWEXBXSAXREX5QJX2Q, seg_pos={0 0}, upload=549888]: Long tail cancellation due to optimal threshold
Dec 31 19:01:14.578161215 (627h32m14.44924425s) [noise=true , address=45.91.116.6:4974 , piece_id=R5FADLJDMRBYMO5AZL3MR6GUA55O4KTGJ6EWCB725JBHXSCMCYHA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:48032->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:48032->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48032->45.91.116.6:4974: use of closed network connection
Dec 31 19:01:14.578167326 (627h32m14.449250361s) [noise=true , address=45.91.116.6:4974 , piece_id=R5FADLJDMRBYMO5AZL3MR6GUA55O4KTGJ6EWCB725JBHXSCMCYHA, seg_pos={0 0}, upload=549888]: Long tail cancellation due to optimal threshold
Dec 31 19:01:14.578181263 (627h32m14.449264308s) [seg_pos={0 0}, upload=549888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 19:01:14.578277015 (627h32m14.44936005s) [upload=549888]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 19:01:14.578281534 (627h32m14.449364589s) [upload=549888]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 19:01:14.875461937 (627h32m14.746544982s) [upload=549888]: Done uploading segments...
Dec 31 19:01:14.875484871 (627h32m14.746567916s) [upload=549888]: Upload finished. err: <nil>
Duration: 8.815848056s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
6.138591628s trace:
https://jaeger.dev.storj.io/trace/9459b27b9ebbd
FAIL (2025-12-31 19:02:33 UTC)
orbiter warning: context deadline exceeded
Duration: 6.138617326s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [......X.....................................................] duration:
3.668719844s trace:
https://jaeger.dev.storj.io/trace/1c0a173ba9d72fc4
FAIL (2025-12-31 19:02:22 UTC)
orbiter warning: context deadline exceeded
Duration: 3.66878613s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [............................X..................X.X..........] duration:
35.137687652s trace:
https://jaeger.dev.storj.io/trace/3fb4961d520cfd75
FAIL (2025-12-31 19:02:02 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12bYqoLXPqxPwKHkGRWx6RSpY8KSnS7QeKafAmtEpGJ5ThmJcDD, Piece ID: ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A) context deadline exceeded; piecestore: (Node ID: 1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, Piece ID: EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ) context deadline exceeded; piecestore: (Node ID: 1zF1SqsUdx5ikfqnG4SfeFukn8ZgX56sC7DGEPtPxZqEicws6o, Piece ID: PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA) context deadline exceeded
======LOGS======
, address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.958820533 (627h24m29.829903569s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.959307350 (627h24m29.830390386s) [noise=true , address=145.79.227.49:4349 , piece_id=R2IQZU7VSWQRVSJITAEMDNNJIPSJA3LBGC2X27AZ7GNZSN4XW2TA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.959608194 (627h24m29.830691229s) [noise=true , address=145.79.227.24:4588 , piece_id=5DP6OWR4UURMV22UA73ZIWUITQCEERVA6KE2K3T6TVDXAV2SO65A, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.959886383 (627h24m29.830969429s) [noise=true , address=64.49.13.26:4945 , piece_id=UMBXDB3DCEJERPVVINHZPPLJ7GKLDZ4PCZKBEOQD5RY5MB7LPCJQ, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.960144104 (627h24m29.831227139s) [noise=true , address=45.91.116.6:4008 , piece_id=64B6RIG5ANWZEVEMNU5HGC6X6E7EHBAIPDJN77UT6RQYYXRGVRWA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.960436030 (627h24m29.831519065s) [noise=true , address=145.79.227.47:4289 , piece_id=GMAPJV5LDECT7ZKYL23XWLDDTYHRAREVFZ2PEIH6OVJNTHUTTAHQ, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.960723447 (627h24m29.831806482s) [noise=true , address=145.79.227.42:4134 , piece_id=SWUYLD4EMMIYVLSWKSO3G3VOQ32ZGAI7NFX7KOURSIN3X6IZ4YJQ, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.960995856 (627h24m29.832078891s) [noise=true , address=145.79.227.40:4084 , piece_id=FD4PW56PCRS3QYHNXA6UQ3SXW2HNFIV3W53XFKMCQMAHDF542N3A, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.961374016 (627h24m29.832457051s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.961713552 (627h24m29.832796587s) [noise=true , address=64.49.13.137:4501 , piece_id=JBYIIB5EY4R2PJULBSLUFDQ3CMMYYIZHTS4BIX7GV5F2KY5OZC6A, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.962169091 (627h24m29.833252136s) [noise=true , address=145.79.227.2:4132 , piece_id=H2G6OZT52HEDBWWOSPU3VS5XIUT64XP5BIO2U7VAX5LNARERBQFA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.962533905 (627h24m29.83361695s) [noise=true , address=145.79.227.18:4027 , piece_id=RVLZBSJIG7UIDS6ITMGFQZJSJEKKDG4G6RFWYLXPUL7AEDZRZHMA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.963210673 (627h24m29.834293708s) [noise=true , address=145.79.227.35:4502 , piece_id=JDBTVMAGPR7M2TPMWJKMDQ562USDDYYPYFQMGEJSTNPARS6NAD5A, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.963755381 (627h24m29.834838426s) [noise=true , address=64.49.13.27:4957 , piece_id=ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.964023922 (627h24m29.835106957s) [noise=true , address=77.90.31.15:4730 , piece_id=PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.964592434 (627h24m29.835675469s) [noise=true , address=77.90.31.16:4753 , piece_id=EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.965227163 (627h24m29.836310208s) [noise=true , address=145.79.227.30:4770 , piece_id=TRAIDZYLZCU5UGHDYAJREYUELZQ5H5AFU4HHPYQFFKPXRZ5OJAQQ, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:31.410300275 (627h24m31.28138331s) [upload=549775]: Next returned nil segment
Dec 31 18:53:31.410315224 (627h24m31.281398259s) [upload=549775]: Waiting for error group managing segments...
Dec 31 18:53:31.788440578 (627h24m31.659523613s) [noise=true , address=64.49.13.26:4945 , piece_id=UMBXDB3DCEJERPVVINHZPPLJ7GKLDZ4PCZKBEOQD5RY5MB7LPCJQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.858179480 (627h24m31.729262526s) [noise=true , address=145.79.227.42:4134 , piece_id=SWUYLD4EMMIYVLSWKSO3G3VOQ32ZGAI7NFX7KOURSIN3X6IZ4YJQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.887866078 (627h24m31.758949113s) [noise=true , address=145.79.227.24:4588 , piece_id=5DP6OWR4UURMV22UA73ZIWUITQCEERVA6KE2K3T6TVDXAV2SO65A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.918774252 (627h24m31.789857297s) [noise=true , address=145.79.227.18:4027 , piece_id=RVLZBSJIG7UIDS6ITMGFQZJSJEKKDG4G6RFWYLXPUL7AEDZRZHMA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.922490298 (627h24m31.793573333s) [noise=true , address=145.79.227.35:4502 , piece_id=JDBTVMAGPR7M2TPMWJKMDQ562USDDYYPYFQMGEJSTNPARS6NAD5A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.924120612 (627h24m31.795203647s) [noise=true , address=145.79.227.2:4132 , piece_id=H2G6OZT52HEDBWWOSPU3VS5XIUT64XP5BIO2U7VAX5LNARERBQFA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.971450444 (627h24m31.842533479s) [noise=true , address=145.79.227.47:4289 , piece_id=GMAPJV5LDECT7ZKYL23XWLDDTYHRAREVFZ2PEIH6OVJNTHUTTAHQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.973634513 (627h24m31.844717548s) [noise=true , address=64.49.13.27:4957 , piece_id=ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.984147871 (627h24m31.855230906s) [noise=true , address=64.49.13.137:4501 , piece_id=JBYIIB5EY4R2PJULBSLUFDQ3CMMYYIZHTS4BIX7GV5F2KY5OZC6A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.002802704 (627h24m31.873885749s) [noise=true , address=77.90.31.15:4730 , piece_id=PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.113876665 (627h24m31.9849597s) [noise=true , address=77.90.31.16:4753 , piece_id=EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.307710471 (627h24m32.178793506s) [noise=true , address=45.91.116.6:4008 , piece_id=64B6RIG5ANWZEVEMNU5HGC6X6E7EHBAIPDJN77UT6RQYYXRGVRWA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.740289560 (627h24m32.611372605s) [noise=true , address=145.79.227.30:4770 , piece_id=TRAIDZYLZCU5UGHDYAJREYUELZQ5H5AFU4HHPYQFFKPXRZ5OJAQQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.744374660 (627h24m32.615457715s) [noise=true , address=145.79.227.49:4349 , piece_id=R2IQZU7VSWQRVSJITAEMDNNJIPSJA3LBGC2X27AZ7GNZSN4XW2TA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.744399677 (627h24m32.615482712s) [seg_pos={0 0}, upload=549775]: Segment reached optimal threshold of 14 pieces.
Dec 31 18:53:32.744624495 (627h24m32.61570753s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA): protocol: write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection; write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection
Dec 31 18:53:32.744633362 (627h24m32.615716397s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold
Dec 31 18:53:32.744938423 (627h24m32.616021468s) [noise=true , address=145.79.227.40:4084 , piece_id=FD4PW56PCRS3QYHNXA6UQ3SXW2HNFIV3W53XFKMCQMAHDF542N3A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.745025308 (627h24m32.616108343s) [noise=true , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gh7eXJMeRFyjBP2e9ZyTQVVE1mMoMFfSkQ9ssfxJyqG63C4gP): protocol: expected piece hash; context canceled; EOF
Dec 31 18:53:32.745035146 (627h24m32.616118181s) [noise=true , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold
Dec 31 18:53:32.745291255 (627h24m32.61637429s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu): protocol: expected piece hash; context canceled; EOF
Dec 31 18:53:32.745299210 (627h24m32.616382245s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold
Dec 31 18:53:32.745319939 (627h24m32.616402985s) [seg_pos={0 0}, upload=549775]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil>
Dec 31 18:53:32.745400913 (627h24m32.616483948s) [upload=549775]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 18:53:32.745406434 (627h24m32.616489479s) [upload=549775]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 18:53:33.044217872 (627h24m32.915300907s) [upload=549775]: Done uploading segments...
Dec 31 18:53:33.044236999 (627h24m32.915320034s) [upload=549775]: Upload finished. err: <nil>
Duration: 35.137722709s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
22 out of 60 is failed (threshold=3) [.XXX.X.X.X.X.X....XX.X....X...X.X...XXX..X..X........X...XX.] duration:
5.98139808s trace:
https://jaeger.dev.storj.io/trace/735677880864da8a
FAIL (2025-12-31 19:01:51 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: CU2OUXDQYV6ULLGUUUC464LIWQS45JTPB3CTMNQZC3JMIYH2SH4Q) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: 4PUJ6YDAY364WVULRTHYBMG6MBLFDWKLLIPEVYFWK6HCQF3PC6LQ) context deadline exceeded
======LOGS======
segments...
Dec 31 19:01:11.769976062 (627h32m11.641059107s) [upload=549888]: Got next segment. Inline: false
Dec 31 19:01:11.770023953 (627h32m11.641106998s) [upload=549888]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 19:01:11.770043050 (627h32m11.641126085s) [upload=549888]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 19:01:12.023231878 (627h32m11.894314913s) [seg_pos={0 0}, upload=549888]: Begin upload segment...
Dec 31 19:01:12.023296881 (627h32m11.894379916s) [seg_pos={0 0}, upload=549888]: Done begin upload segment.
Dec 31 19:01:12.023355753 (627h32m11.894438798s) [noise=true , address=77.90.31.10:4519 , piece_id=C25QYCH2VTZ6PE72D467NNLTBB74MU74XJQ36CQYTWO5TG22HG7A, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.023376493 (627h32m11.894459528s) [noise=true , address=77.90.31.14:4662 , piece_id=CU2OUXDQYV6ULLGUUUC464LIWQS45JTPB3CTMNQZC3JMIYH2SH4Q, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.023952419 (627h32m11.895035454s) [noise=true , address=45.91.116.11:4119 , piece_id=LDYC6K6PPHDKOWGNEQXU464PFIBYSPWXMJZBEY7ELBRPP5YL57EA, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.024015850 (627h32m11.895098885s) [noise=true , address=77.90.31.16:4736 , piece_id=PYWN35W6PGUC3NMTIROYVLH7UY2TJH3LRJWWEXBXSAXREX5QJX2Q, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.024311754 (627h32m11.895394789s) [noise=true , address=145.79.227.46:4603 , piece_id=RTKSKQ4L63I7MBTLOAMZE36DNSPQ2ZPXV64KRMQPNWCHHODZIYGA, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.024431361 (627h32m11.895514397s) [noise=true , address=145.79.227.6:4259 , piece_id=OQAYR6PIVUKBG2A3OKZBZX3H7ZJFEUR256I3IU6QWYQ4X7PVZGSQ, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.024577730 (627h32m11.895660765s) [noise=true , address=145.79.227.14:4840 , piece_id=QYG4BKZ54RFDZGVXEYNGXRAL4INUYCE4CLPWKCETGL7DSBOKEFKA, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.024765618 (627h32m11.895848653s) [noise=true , address=45.91.116.14:4189 , piece_id=4PUJ6YDAY364WVULRTHYBMG6MBLFDWKLLIPEVYFWK6HCQF3PC6LQ, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.024809993 (627h32m11.895893028s) [noise=true , address=145.79.227.22:4509 , piece_id=WVFB7RFZSVW2M446KJGVIJ4FUGYC7QLEAB6TYY6J2O563KXBFT6A, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.025153747 (627h32m11.896236782s) [noise=true , address=77.90.31.19:4842 , piece_id=ZE2C5YWUKVXYLJTMKQRFYN6MVZ6EUOP4DLU4FEB6D6F3VXII7ZYA, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.025220134 (627h32m11.896303179s) [noise=true , address=45.91.116.6:4974 , piece_id=R5FADLJDMRBYMO5AZL3MR6GUA55O4KTGJ6EWCB725JBHXSCMCYHA, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.025492613 (627h32m11.896575648s) [noise=true , address=77.90.31.18:4805 , piece_id=ZHDEUEKMLONGWMVC2FXD5OGIYUKO6QF5BSORJTSESP4JUJP7V2EQ, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.025786592 (627h32m11.896869627s) [noise=true , address=45.91.116.13:4156 , piece_id=LOJNZHUI2LDUHLQMKM2ZBRNRKVDEX7UXSBUR3WZJNOSCU36GVEUA, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.026165153 (627h32m11.897248198s) [noise=true , address=145.79.227.48:4303 , piece_id=HGCHG7IT4ZBWBYCPULLYDC6M2CAOCNR4PSIH4JU3Y5WP7FOTYUZA, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.026497455 (627h32m11.8975805s) [noise=true , address=45.91.116.12:4228 , piece_id=I5N63CF36AUCN2OGF66OV5GEVOQYQSUR3CS6YLJWWVMNGUL2W3XA, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.026680595 (627h32m11.89776364s) [noise=true , address=77.90.31.21:4878 , piece_id=FX5WINCM4HKZVSKP6MTEKZOMOEWJRJ4J7E4HVGLG3CQSEDJXJLSQ, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:13.730039170 (627h32m13.601122215s) [upload=549888]: Next returned nil segment
Dec 31 19:01:13.730056022 (627h32m13.601139057s) [upload=549888]: Waiting for error group managing segments...
Dec 31 19:01:14.011727154 (627h32m13.882810199s) [noise=true , address=145.79.227.46:4603 , piece_id=RTKSKQ4L63I7MBTLOAMZE36DNSPQ2ZPXV64KRMQPNWCHHODZIYGA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.015213612 (627h32m13.886296647s) [noise=true , address=145.79.227.22:4509 , piece_id=WVFB7RFZSVW2M446KJGVIJ4FUGYC7QLEAB6TYY6J2O563KXBFT6A, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.016168130 (627h32m13.887251165s) [noise=true , address=145.79.227.6:4259 , piece_id=OQAYR6PIVUKBG2A3OKZBZX3H7ZJFEUR256I3IU6QWYQ4X7PVZGSQ, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.067093519 (627h32m13.938176554s) [noise=true , address=145.79.227.14:4840 , piece_id=QYG4BKZ54RFDZGVXEYNGXRAL4INUYCE4CLPWKCETGL7DSBOKEFKA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.094476500 (627h32m13.965559545s) [noise=true , address=145.79.227.48:4303 , piece_id=HGCHG7IT4ZBWBYCPULLYDC6M2CAOCNR4PSIH4JU3Y5WP7FOTYUZA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.197378931 (627h32m14.068461976s) [noise=true , address=77.90.31.14:4662 , piece_id=CU2OUXDQYV6ULLGUUUC464LIWQS45JTPB3CTMNQZC3JMIYH2SH4Q, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.260600915 (627h32m14.13168395s) [noise=true , address=77.90.31.10:4519 , piece_id=C25QYCH2VTZ6PE72D467NNLTBB74MU74XJQ36CQYTWO5TG22HG7A, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.292833670 (627h32m14.163916715s) [noise=true , address=77.90.31.19:4842 , piece_id=ZE2C5YWUKVXYLJTMKQRFYN6MVZ6EUOP4DLU4FEB6D6F3VXII7ZYA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.335927357 (627h32m14.207010392s) [noise=true , address=45.91.116.11:4119 , piece_id=LDYC6K6PPHDKOWGNEQXU464PFIBYSPWXMJZBEY7ELBRPP5YL57EA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.363700461 (627h32m14.234783496s) [noise=true , address=45.91.116.13:4156 , piece_id=LOJNZHUI2LDUHLQMKM2ZBRNRKVDEX7UXSBUR3WZJNOSCU36GVEUA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.375270873 (627h32m14.246353918s) [noise=true , address=77.90.31.21:4878 , piece_id=FX5WINCM4HKZVSKP6MTEKZOMOEWJRJ4J7E4HVGLG3CQSEDJXJLSQ, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.458059915 (627h32m14.32914295s) [noise=true , address=45.91.116.14:4189 , piece_id=4PUJ6YDAY364WVULRTHYBMG6MBLFDWKLLIPEVYFWK6HCQF3PC6LQ, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.577656003 (627h32m14.448739038s) [noise=true , address=77.90.31.18:4805 , piece_id=ZHDEUEKMLONGWMVC2FXD5OGIYUKO6QF5BSORJTSESP4JUJP7V2EQ, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.577676772 (627h32m14.448759807s) [seg_pos={0 0}, upload=549888]: Segment reached optimal threshold of 13 pieces.
Dec 31 19:01:14.578054592 (627h32m14.449137627s) [noise=true , address=45.91.116.12:4228 , piece_id=I5N63CF36AUCN2OGF66OV5GEVOQYQSUR3CS6YLJWWVMNGUL2W3XA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:47206->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:47206->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47206->45.91.116.12:4228: use of closed network connection
Dec 31 19:01:14.578062086 (627h32m14.449145121s) [noise=true , address=45.91.116.12:4228 , piece_id=I5N63CF36AUCN2OGF66OV5GEVOQYQSUR3CS6YLJWWVMNGUL2W3XA, seg_pos={0 0}, upload=549888]: Long tail cancellation due to optimal threshold
Dec 31 19:01:14.578143692 (627h32m14.449226747s) [noise=true , address=77.90.31.16:4736 , piece_id=PYWN35W6PGUC3NMTIROYVLH7UY2TJH3LRJWWEXBXSAXREX5QJX2Q, seg_pos={0 0}, upload=549888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF
Dec 31 19:01:14.578156606 (627h32m14.449239671s) [noise=true , address=77.90.31.16:4736 , piece_id=PYWN35W6PGUC3NMTIROYVLH7UY2TJH3LRJWWEXBXSAXREX5QJX2Q, seg_pos={0 0}, upload=549888]: Long tail cancellation due to optimal threshold
Dec 31 19:01:14.578161215 (627h32m14.44924425s) [noise=true , address=45.91.116.6:4974 , piece_id=R5FADLJDMRBYMO5AZL3MR6GUA55O4KTGJ6EWCB725JBHXSCMCYHA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:48032->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:48032->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48032->45.91.116.6:4974: use of closed network connection
Dec 31 19:01:14.578167326 (627h32m14.449250361s) [noise=true , address=45.91.116.6:4974 , piece_id=R5FADLJDMRBYMO5AZL3MR6GUA55O4KTGJ6EWCB725JBHXSCMCYHA, seg_pos={0 0}, upload=549888]: Long tail cancellation due to optimal threshold
Dec 31 19:01:14.578181263 (627h32m14.449264308s) [seg_pos={0 0}, upload=549888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 19:01:14.578277015 (627h32m14.44936005s) [upload=549888]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 19:01:14.578281534 (627h32m14.449364589s) [upload=549888]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 19:01:14.875461937 (627h32m14.746544982s) [upload=549888]: Done uploading segments...
Dec 31 19:01:14.875484871 (627h32m14.746567916s) [upload=549888]: Upload finished. err: <nil>
Duration: 5.981425915s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
22 out of 60 is failed (threshold=3) [..XXX.X.X.X.X.X....XX.X....X...X.X...XXX..X..X........X...XX] duration:
30.0018821s trace:
https://jaeger.dev.storj.io/trace/735677880864da8a
FAIL (2025-12-31 19:01:41 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: CU2OUXDQYV6ULLGUUUC464LIWQS45JTPB3CTMNQZC3JMIYH2SH4Q) context deadline exceeded; piecestore: (Node ID: 1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N, Piece ID: 4PUJ6YDAY364WVULRTHYBMG6MBLFDWKLLIPEVYFWK6HCQF3PC6LQ) context deadline exceeded
======LOGS======
segments...
Dec 31 19:01:11.769976062 (627h32m11.641059107s) [upload=549888]: Got next segment. Inline: false
Dec 31 19:01:11.770023953 (627h32m11.641106998s) [upload=549888]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 19:01:11.770043050 (627h32m11.641126085s) [upload=549888]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 19:01:12.023231878 (627h32m11.894314913s) [seg_pos={0 0}, upload=549888]: Begin upload segment...
Dec 31 19:01:12.023296881 (627h32m11.894379916s) [seg_pos={0 0}, upload=549888]: Done begin upload segment.
Dec 31 19:01:12.023355753 (627h32m11.894438798s) [noise=true , address=77.90.31.10:4519 , piece_id=C25QYCH2VTZ6PE72D467NNLTBB74MU74XJQ36CQYTWO5TG22HG7A, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.023376493 (627h32m11.894459528s) [noise=true , address=77.90.31.14:4662 , piece_id=CU2OUXDQYV6ULLGUUUC464LIWQS45JTPB3CTMNQZC3JMIYH2SH4Q, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.023952419 (627h32m11.895035454s) [noise=true , address=45.91.116.11:4119 , piece_id=LDYC6K6PPHDKOWGNEQXU464PFIBYSPWXMJZBEY7ELBRPP5YL57EA, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.024015850 (627h32m11.895098885s) [noise=true , address=77.90.31.16:4736 , piece_id=PYWN35W6PGUC3NMTIROYVLH7UY2TJH3LRJWWEXBXSAXREX5QJX2Q, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.024311754 (627h32m11.895394789s) [noise=true , address=145.79.227.46:4603 , piece_id=RTKSKQ4L63I7MBTLOAMZE36DNSPQ2ZPXV64KRMQPNWCHHODZIYGA, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.024431361 (627h32m11.895514397s) [noise=true , address=145.79.227.6:4259 , piece_id=OQAYR6PIVUKBG2A3OKZBZX3H7ZJFEUR256I3IU6QWYQ4X7PVZGSQ, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.024577730 (627h32m11.895660765s) [noise=true , address=145.79.227.14:4840 , piece_id=QYG4BKZ54RFDZGVXEYNGXRAL4INUYCE4CLPWKCETGL7DSBOKEFKA, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.024765618 (627h32m11.895848653s) [noise=true , address=45.91.116.14:4189 , piece_id=4PUJ6YDAY364WVULRTHYBMG6MBLFDWKLLIPEVYFWK6HCQF3PC6LQ, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.024809993 (627h32m11.895893028s) [noise=true , address=145.79.227.22:4509 , piece_id=WVFB7RFZSVW2M446KJGVIJ4FUGYC7QLEAB6TYY6J2O563KXBFT6A, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.025153747 (627h32m11.896236782s) [noise=true , address=77.90.31.19:4842 , piece_id=ZE2C5YWUKVXYLJTMKQRFYN6MVZ6EUOP4DLU4FEB6D6F3VXII7ZYA, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.025220134 (627h32m11.896303179s) [noise=true , address=45.91.116.6:4974 , piece_id=R5FADLJDMRBYMO5AZL3MR6GUA55O4KTGJ6EWCB725JBHXSCMCYHA, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.025492613 (627h32m11.896575648s) [noise=true , address=77.90.31.18:4805 , piece_id=ZHDEUEKMLONGWMVC2FXD5OGIYUKO6QF5BSORJTSESP4JUJP7V2EQ, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.025786592 (627h32m11.896869627s) [noise=true , address=45.91.116.13:4156 , piece_id=LOJNZHUI2LDUHLQMKM2ZBRNRKVDEX7UXSBUR3WZJNOSCU36GVEUA, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.026165153 (627h32m11.897248198s) [noise=true , address=145.79.227.48:4303 , piece_id=HGCHG7IT4ZBWBYCPULLYDC6M2CAOCNR4PSIH4JU3Y5WP7FOTYUZA, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.026497455 (627h32m11.8975805s) [noise=true , address=45.91.116.12:4228 , piece_id=I5N63CF36AUCN2OGF66OV5GEVOQYQSUR3CS6YLJWWVMNGUL2W3XA, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:12.026680595 (627h32m11.89776364s) [noise=true , address=77.90.31.21:4878 , piece_id=FX5WINCM4HKZVSKP6MTEKZOMOEWJRJ4J7E4HVGLG3CQSEDJXJLSQ, seg_pos={0 0}, upload=549888]: Uploading piece...
Dec 31 19:01:13.730039170 (627h32m13.601122215s) [upload=549888]: Next returned nil segment
Dec 31 19:01:13.730056022 (627h32m13.601139057s) [upload=549888]: Waiting for error group managing segments...
Dec 31 19:01:14.011727154 (627h32m13.882810199s) [noise=true , address=145.79.227.46:4603 , piece_id=RTKSKQ4L63I7MBTLOAMZE36DNSPQ2ZPXV64KRMQPNWCHHODZIYGA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.015213612 (627h32m13.886296647s) [noise=true , address=145.79.227.22:4509 , piece_id=WVFB7RFZSVW2M446KJGVIJ4FUGYC7QLEAB6TYY6J2O563KXBFT6A, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.016168130 (627h32m13.887251165s) [noise=true , address=145.79.227.6:4259 , piece_id=OQAYR6PIVUKBG2A3OKZBZX3H7ZJFEUR256I3IU6QWYQ4X7PVZGSQ, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.067093519 (627h32m13.938176554s) [noise=true , address=145.79.227.14:4840 , piece_id=QYG4BKZ54RFDZGVXEYNGXRAL4INUYCE4CLPWKCETGL7DSBOKEFKA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.094476500 (627h32m13.965559545s) [noise=true , address=145.79.227.48:4303 , piece_id=HGCHG7IT4ZBWBYCPULLYDC6M2CAOCNR4PSIH4JU3Y5WP7FOTYUZA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.197378931 (627h32m14.068461976s) [noise=true , address=77.90.31.14:4662 , piece_id=CU2OUXDQYV6ULLGUUUC464LIWQS45JTPB3CTMNQZC3JMIYH2SH4Q, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.260600915 (627h32m14.13168395s) [noise=true , address=77.90.31.10:4519 , piece_id=C25QYCH2VTZ6PE72D467NNLTBB74MU74XJQ36CQYTWO5TG22HG7A, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.292833670 (627h32m14.163916715s) [noise=true , address=77.90.31.19:4842 , piece_id=ZE2C5YWUKVXYLJTMKQRFYN6MVZ6EUOP4DLU4FEB6D6F3VXII7ZYA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.335927357 (627h32m14.207010392s) [noise=true , address=45.91.116.11:4119 , piece_id=LDYC6K6PPHDKOWGNEQXU464PFIBYSPWXMJZBEY7ELBRPP5YL57EA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.363700461 (627h32m14.234783496s) [noise=true , address=45.91.116.13:4156 , piece_id=LOJNZHUI2LDUHLQMKM2ZBRNRKVDEX7UXSBUR3WZJNOSCU36GVEUA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.375270873 (627h32m14.246353918s) [noise=true , address=77.90.31.21:4878 , piece_id=FX5WINCM4HKZVSKP6MTEKZOMOEWJRJ4J7E4HVGLG3CQSEDJXJLSQ, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.458059915 (627h32m14.32914295s) [noise=true , address=45.91.116.14:4189 , piece_id=4PUJ6YDAY364WVULRTHYBMG6MBLFDWKLLIPEVYFWK6HCQF3PC6LQ, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.577656003 (627h32m14.448739038s) [noise=true , address=77.90.31.18:4805 , piece_id=ZHDEUEKMLONGWMVC2FXD5OGIYUKO6QF5BSORJTSESP4JUJP7V2EQ, seg_pos={0 0}, upload=549888]: Done uploading piece. err: <nil>
Dec 31 19:01:14.577676772 (627h32m14.448759807s) [seg_pos={0 0}, upload=549888]: Segment reached optimal threshold of 13 pieces.
Dec 31 19:01:14.578054592 (627h32m14.449137627s) [noise=true , address=45.91.116.12:4228 , piece_id=I5N63CF36AUCN2OGF66OV5GEVOQYQSUR3CS6YLJWWVMNGUL2W3XA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:47206->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:47206->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47206->45.91.116.12:4228: use of closed network connection
Dec 31 19:01:14.578062086 (627h32m14.449145121s) [noise=true , address=45.91.116.12:4228 , piece_id=I5N63CF36AUCN2OGF66OV5GEVOQYQSUR3CS6YLJWWVMNGUL2W3XA, seg_pos={0 0}, upload=549888]: Long tail cancellation due to optimal threshold
Dec 31 19:01:14.578143692 (627h32m14.449226747s) [noise=true , address=77.90.31.16:4736 , piece_id=PYWN35W6PGUC3NMTIROYVLH7UY2TJH3LRJWWEXBXSAXREX5QJX2Q, seg_pos={0 0}, upload=549888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF
Dec 31 19:01:14.578156606 (627h32m14.449239671s) [noise=true , address=77.90.31.16:4736 , piece_id=PYWN35W6PGUC3NMTIROYVLH7UY2TJH3LRJWWEXBXSAXREX5QJX2Q, seg_pos={0 0}, upload=549888]: Long tail cancellation due to optimal threshold
Dec 31 19:01:14.578161215 (627h32m14.44924425s) [noise=true , address=45.91.116.6:4974 , piece_id=R5FADLJDMRBYMO5AZL3MR6GUA55O4KTGJ6EWCB725JBHXSCMCYHA, seg_pos={0 0}, upload=549888]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ): protocol: write tcp 188.166.225.170:48032->45.91.116.6:4974: use of closed network connection; write tcp 188.166.225.170:48032->45.91.116.6:4974: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48032->45.91.116.6:4974: use of closed network connection
Dec 31 19:01:14.578167326 (627h32m14.449250361s) [noise=true , address=45.91.116.6:4974 , piece_id=R5FADLJDMRBYMO5AZL3MR6GUA55O4KTGJ6EWCB725JBHXSCMCYHA, seg_pos={0 0}, upload=549888]: Long tail cancellation due to optimal threshold
Dec 31 19:01:14.578181263 (627h32m14.449264308s) [seg_pos={0 0}, upload=549888]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 19:01:14.578277015 (627h32m14.44936005s) [upload=549888]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 19:01:14.578281534 (627h32m14.449364589s) [upload=549888]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 19:01:14.875461937 (627h32m14.746544982s) [upload=549888]: Done uploading segments...
Dec 31 19:01:14.875484871 (627h32m14.746567916s) [upload=549888]: Upload finished. err: <nil>
Duration: 30.00190854s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.......X....................................................] duration:
3.525350576s trace:
https://jaeger.dev.storj.io/trace/1c0a173ba9d72fc4
FAIL (2025-12-31 19:01:41 UTC)
orbiter warning: context deadline exceeded
Duration: 3.525378618s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
6.265596711s trace:
https://jaeger.dev.storj.io/trace/9459b27b9ebbd
FAIL (2025-12-31 19:01:28 UTC)
orbiter warning: context deadline exceeded
Duration: 6.265631357s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [.............................X..................X.X.........] duration:
27.514256227s trace:
https://jaeger.dev.storj.io/trace/3fb4961d520cfd75
FAIL (2025-12-31 19:01:27 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12bYqoLXPqxPwKHkGRWx6RSpY8KSnS7QeKafAmtEpGJ5ThmJcDD, Piece ID: ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A) context deadline exceeded; piecestore: (Node ID: 1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, Piece ID: EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ) context deadline exceeded; piecestore: (Node ID: 1zF1SqsUdx5ikfqnG4SfeFukn8ZgX56sC7DGEPtPxZqEicws6o, Piece ID: PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA) context deadline exceeded
======LOGS======
, address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.958820533 (627h24m29.829903569s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.959307350 (627h24m29.830390386s) [noise=true , address=145.79.227.49:4349 , piece_id=R2IQZU7VSWQRVSJITAEMDNNJIPSJA3LBGC2X27AZ7GNZSN4XW2TA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.959608194 (627h24m29.830691229s) [noise=true , address=145.79.227.24:4588 , piece_id=5DP6OWR4UURMV22UA73ZIWUITQCEERVA6KE2K3T6TVDXAV2SO65A, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.959886383 (627h24m29.830969429s) [noise=true , address=64.49.13.26:4945 , piece_id=UMBXDB3DCEJERPVVINHZPPLJ7GKLDZ4PCZKBEOQD5RY5MB7LPCJQ, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.960144104 (627h24m29.831227139s) [noise=true , address=45.91.116.6:4008 , piece_id=64B6RIG5ANWZEVEMNU5HGC6X6E7EHBAIPDJN77UT6RQYYXRGVRWA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.960436030 (627h24m29.831519065s) [noise=true , address=145.79.227.47:4289 , piece_id=GMAPJV5LDECT7ZKYL23XWLDDTYHRAREVFZ2PEIH6OVJNTHUTTAHQ, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.960723447 (627h24m29.831806482s) [noise=true , address=145.79.227.42:4134 , piece_id=SWUYLD4EMMIYVLSWKSO3G3VOQ32ZGAI7NFX7KOURSIN3X6IZ4YJQ, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.960995856 (627h24m29.832078891s) [noise=true , address=145.79.227.40:4084 , piece_id=FD4PW56PCRS3QYHNXA6UQ3SXW2HNFIV3W53XFKMCQMAHDF542N3A, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.961374016 (627h24m29.832457051s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.961713552 (627h24m29.832796587s) [noise=true , address=64.49.13.137:4501 , piece_id=JBYIIB5EY4R2PJULBSLUFDQ3CMMYYIZHTS4BIX7GV5F2KY5OZC6A, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.962169091 (627h24m29.833252136s) [noise=true , address=145.79.227.2:4132 , piece_id=H2G6OZT52HEDBWWOSPU3VS5XIUT64XP5BIO2U7VAX5LNARERBQFA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.962533905 (627h24m29.83361695s) [noise=true , address=145.79.227.18:4027 , piece_id=RVLZBSJIG7UIDS6ITMGFQZJSJEKKDG4G6RFWYLXPUL7AEDZRZHMA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.963210673 (627h24m29.834293708s) [noise=true , address=145.79.227.35:4502 , piece_id=JDBTVMAGPR7M2TPMWJKMDQ562USDDYYPYFQMGEJSTNPARS6NAD5A, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.963755381 (627h24m29.834838426s) [noise=true , address=64.49.13.27:4957 , piece_id=ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.964023922 (627h24m29.835106957s) [noise=true , address=77.90.31.15:4730 , piece_id=PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.964592434 (627h24m29.835675469s) [noise=true , address=77.90.31.16:4753 , piece_id=EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.965227163 (627h24m29.836310208s) [noise=true , address=145.79.227.30:4770 , piece_id=TRAIDZYLZCU5UGHDYAJREYUELZQ5H5AFU4HHPYQFFKPXRZ5OJAQQ, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:31.410300275 (627h24m31.28138331s) [upload=549775]: Next returned nil segment
Dec 31 18:53:31.410315224 (627h24m31.281398259s) [upload=549775]: Waiting for error group managing segments...
Dec 31 18:53:31.788440578 (627h24m31.659523613s) [noise=true , address=64.49.13.26:4945 , piece_id=UMBXDB3DCEJERPVVINHZPPLJ7GKLDZ4PCZKBEOQD5RY5MB7LPCJQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.858179480 (627h24m31.729262526s) [noise=true , address=145.79.227.42:4134 , piece_id=SWUYLD4EMMIYVLSWKSO3G3VOQ32ZGAI7NFX7KOURSIN3X6IZ4YJQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.887866078 (627h24m31.758949113s) [noise=true , address=145.79.227.24:4588 , piece_id=5DP6OWR4UURMV22UA73ZIWUITQCEERVA6KE2K3T6TVDXAV2SO65A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.918774252 (627h24m31.789857297s) [noise=true , address=145.79.227.18:4027 , piece_id=RVLZBSJIG7UIDS6ITMGFQZJSJEKKDG4G6RFWYLXPUL7AEDZRZHMA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.922490298 (627h24m31.793573333s) [noise=true , address=145.79.227.35:4502 , piece_id=JDBTVMAGPR7M2TPMWJKMDQ562USDDYYPYFQMGEJSTNPARS6NAD5A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.924120612 (627h24m31.795203647s) [noise=true , address=145.79.227.2:4132 , piece_id=H2G6OZT52HEDBWWOSPU3VS5XIUT64XP5BIO2U7VAX5LNARERBQFA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.971450444 (627h24m31.842533479s) [noise=true , address=145.79.227.47:4289 , piece_id=GMAPJV5LDECT7ZKYL23XWLDDTYHRAREVFZ2PEIH6OVJNTHUTTAHQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.973634513 (627h24m31.844717548s) [noise=true , address=64.49.13.27:4957 , piece_id=ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.984147871 (627h24m31.855230906s) [noise=true , address=64.49.13.137:4501 , piece_id=JBYIIB5EY4R2PJULBSLUFDQ3CMMYYIZHTS4BIX7GV5F2KY5OZC6A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.002802704 (627h24m31.873885749s) [noise=true , address=77.90.31.15:4730 , piece_id=PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.113876665 (627h24m31.9849597s) [noise=true , address=77.90.31.16:4753 , piece_id=EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.307710471 (627h24m32.178793506s) [noise=true , address=45.91.116.6:4008 , piece_id=64B6RIG5ANWZEVEMNU5HGC6X6E7EHBAIPDJN77UT6RQYYXRGVRWA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.740289560 (627h24m32.611372605s) [noise=true , address=145.79.227.30:4770 , piece_id=TRAIDZYLZCU5UGHDYAJREYUELZQ5H5AFU4HHPYQFFKPXRZ5OJAQQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.744374660 (627h24m32.615457715s) [noise=true , address=145.79.227.49:4349 , piece_id=R2IQZU7VSWQRVSJITAEMDNNJIPSJA3LBGC2X27AZ7GNZSN4XW2TA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.744399677 (627h24m32.615482712s) [seg_pos={0 0}, upload=549775]: Segment reached optimal threshold of 14 pieces.
Dec 31 18:53:32.744624495 (627h24m32.61570753s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA): protocol: write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection; write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection
Dec 31 18:53:32.744633362 (627h24m32.615716397s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold
Dec 31 18:53:32.744938423 (627h24m32.616021468s) [noise=true , address=145.79.227.40:4084 , piece_id=FD4PW56PCRS3QYHNXA6UQ3SXW2HNFIV3W53XFKMCQMAHDF542N3A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.745025308 (627h24m32.616108343s) [noise=true , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gh7eXJMeRFyjBP2e9ZyTQVVE1mMoMFfSkQ9ssfxJyqG63C4gP): protocol: expected piece hash; context canceled; EOF
Dec 31 18:53:32.745035146 (627h24m32.616118181s) [noise=true , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold
Dec 31 18:53:32.745291255 (627h24m32.61637429s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu): protocol: expected piece hash; context canceled; EOF
Dec 31 18:53:32.745299210 (627h24m32.616382245s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold
Dec 31 18:53:32.745319939 (627h24m32.616402985s) [seg_pos={0 0}, upload=549775]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil>
Dec 31 18:53:32.745400913 (627h24m32.616483948s) [upload=549775]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 18:53:32.745406434 (627h24m32.616489479s) [upload=549775]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 18:53:33.044217872 (627h24m32.915300907s) [upload=549775]: Done uploading segments...
Dec 31 18:53:33.044236999 (627h24m32.915320034s) [upload=549775]: Upload finished. err: <nil>
Duration: 27.514276565s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [........X...................................................] duration:
3.424236672s trace:
https://jaeger.dev.storj.io/trace/1c0a173ba9d72fc4
FAIL (2025-12-31 19:01:13 UTC)
orbiter warning: context deadline exceeded
Duration: 3.424982471s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
6.244164776s trace:
https://jaeger.dev.storj.io/trace/9459b27b9ebbd
FAIL (2025-12-31 19:01:00 UTC)
orbiter warning: context deadline exceeded
Duration: 6.244196116s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [..............................X..................X.X........] duration:
46.461439695s trace:
https://jaeger.dev.storj.io/trace/3fb4961d520cfd75
FAIL (2025-12-31 19:00:59 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12bYqoLXPqxPwKHkGRWx6RSpY8KSnS7QeKafAmtEpGJ5ThmJcDD, Piece ID: ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A) context deadline exceeded; piecestore: (Node ID: 1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, Piece ID: EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ) context deadline exceeded; piecestore: (Node ID: 1zF1SqsUdx5ikfqnG4SfeFukn8ZgX56sC7DGEPtPxZqEicws6o, Piece ID: PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA) context deadline exceeded
======LOGS======
, address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.958820533 (627h24m29.829903569s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.959307350 (627h24m29.830390386s) [noise=true , address=145.79.227.49:4349 , piece_id=R2IQZU7VSWQRVSJITAEMDNNJIPSJA3LBGC2X27AZ7GNZSN4XW2TA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.959608194 (627h24m29.830691229s) [noise=true , address=145.79.227.24:4588 , piece_id=5DP6OWR4UURMV22UA73ZIWUITQCEERVA6KE2K3T6TVDXAV2SO65A, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.959886383 (627h24m29.830969429s) [noise=true , address=64.49.13.26:4945 , piece_id=UMBXDB3DCEJERPVVINHZPPLJ7GKLDZ4PCZKBEOQD5RY5MB7LPCJQ, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.960144104 (627h24m29.831227139s) [noise=true , address=45.91.116.6:4008 , piece_id=64B6RIG5ANWZEVEMNU5HGC6X6E7EHBAIPDJN77UT6RQYYXRGVRWA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.960436030 (627h24m29.831519065s) [noise=true , address=145.79.227.47:4289 , piece_id=GMAPJV5LDECT7ZKYL23XWLDDTYHRAREVFZ2PEIH6OVJNTHUTTAHQ, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.960723447 (627h24m29.831806482s) [noise=true , address=145.79.227.42:4134 , piece_id=SWUYLD4EMMIYVLSWKSO3G3VOQ32ZGAI7NFX7KOURSIN3X6IZ4YJQ, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.960995856 (627h24m29.832078891s) [noise=true , address=145.79.227.40:4084 , piece_id=FD4PW56PCRS3QYHNXA6UQ3SXW2HNFIV3W53XFKMCQMAHDF542N3A, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.961374016 (627h24m29.832457051s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.961713552 (627h24m29.832796587s) [noise=true , address=64.49.13.137:4501 , piece_id=JBYIIB5EY4R2PJULBSLUFDQ3CMMYYIZHTS4BIX7GV5F2KY5OZC6A, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.962169091 (627h24m29.833252136s) [noise=true , address=145.79.227.2:4132 , piece_id=H2G6OZT52HEDBWWOSPU3VS5XIUT64XP5BIO2U7VAX5LNARERBQFA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.962533905 (627h24m29.83361695s) [noise=true , address=145.79.227.18:4027 , piece_id=RVLZBSJIG7UIDS6ITMGFQZJSJEKKDG4G6RFWYLXPUL7AEDZRZHMA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.963210673 (627h24m29.834293708s) [noise=true , address=145.79.227.35:4502 , piece_id=JDBTVMAGPR7M2TPMWJKMDQ562USDDYYPYFQMGEJSTNPARS6NAD5A, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.963755381 (627h24m29.834838426s) [noise=true , address=64.49.13.27:4957 , piece_id=ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.964023922 (627h24m29.835106957s) [noise=true , address=77.90.31.15:4730 , piece_id=PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.964592434 (627h24m29.835675469s) [noise=true , address=77.90.31.16:4753 , piece_id=EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.965227163 (627h24m29.836310208s) [noise=true , address=145.79.227.30:4770 , piece_id=TRAIDZYLZCU5UGHDYAJREYUELZQ5H5AFU4HHPYQFFKPXRZ5OJAQQ, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:31.410300275 (627h24m31.28138331s) [upload=549775]: Next returned nil segment
Dec 31 18:53:31.410315224 (627h24m31.281398259s) [upload=549775]: Waiting for error group managing segments...
Dec 31 18:53:31.788440578 (627h24m31.659523613s) [noise=true , address=64.49.13.26:4945 , piece_id=UMBXDB3DCEJERPVVINHZPPLJ7GKLDZ4PCZKBEOQD5RY5MB7LPCJQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.858179480 (627h24m31.729262526s) [noise=true , address=145.79.227.42:4134 , piece_id=SWUYLD4EMMIYVLSWKSO3G3VOQ32ZGAI7NFX7KOURSIN3X6IZ4YJQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.887866078 (627h24m31.758949113s) [noise=true , address=145.79.227.24:4588 , piece_id=5DP6OWR4UURMV22UA73ZIWUITQCEERVA6KE2K3T6TVDXAV2SO65A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.918774252 (627h24m31.789857297s) [noise=true , address=145.79.227.18:4027 , piece_id=RVLZBSJIG7UIDS6ITMGFQZJSJEKKDG4G6RFWYLXPUL7AEDZRZHMA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.922490298 (627h24m31.793573333s) [noise=true , address=145.79.227.35:4502 , piece_id=JDBTVMAGPR7M2TPMWJKMDQ562USDDYYPYFQMGEJSTNPARS6NAD5A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.924120612 (627h24m31.795203647s) [noise=true , address=145.79.227.2:4132 , piece_id=H2G6OZT52HEDBWWOSPU3VS5XIUT64XP5BIO2U7VAX5LNARERBQFA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.971450444 (627h24m31.842533479s) [noise=true , address=145.79.227.47:4289 , piece_id=GMAPJV5LDECT7ZKYL23XWLDDTYHRAREVFZ2PEIH6OVJNTHUTTAHQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.973634513 (627h24m31.844717548s) [noise=true , address=64.49.13.27:4957 , piece_id=ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.984147871 (627h24m31.855230906s) [noise=true , address=64.49.13.137:4501 , piece_id=JBYIIB5EY4R2PJULBSLUFDQ3CMMYYIZHTS4BIX7GV5F2KY5OZC6A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.002802704 (627h24m31.873885749s) [noise=true , address=77.90.31.15:4730 , piece_id=PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.113876665 (627h24m31.9849597s) [noise=true , address=77.90.31.16:4753 , piece_id=EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.307710471 (627h24m32.178793506s) [noise=true , address=45.91.116.6:4008 , piece_id=64B6RIG5ANWZEVEMNU5HGC6X6E7EHBAIPDJN77UT6RQYYXRGVRWA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.740289560 (627h24m32.611372605s) [noise=true , address=145.79.227.30:4770 , piece_id=TRAIDZYLZCU5UGHDYAJREYUELZQ5H5AFU4HHPYQFFKPXRZ5OJAQQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.744374660 (627h24m32.615457715s) [noise=true , address=145.79.227.49:4349 , piece_id=R2IQZU7VSWQRVSJITAEMDNNJIPSJA3LBGC2X27AZ7GNZSN4XW2TA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.744399677 (627h24m32.615482712s) [seg_pos={0 0}, upload=549775]: Segment reached optimal threshold of 14 pieces.
Dec 31 18:53:32.744624495 (627h24m32.61570753s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA): protocol: write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection; write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection
Dec 31 18:53:32.744633362 (627h24m32.615716397s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold
Dec 31 18:53:32.744938423 (627h24m32.616021468s) [noise=true , address=145.79.227.40:4084 , piece_id=FD4PW56PCRS3QYHNXA6UQ3SXW2HNFIV3W53XFKMCQMAHDF542N3A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.745025308 (627h24m32.616108343s) [noise=true , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gh7eXJMeRFyjBP2e9ZyTQVVE1mMoMFfSkQ9ssfxJyqG63C4gP): protocol: expected piece hash; context canceled; EOF
Dec 31 18:53:32.745035146 (627h24m32.616118181s) [noise=true , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold
Dec 31 18:53:32.745291255 (627h24m32.61637429s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu): protocol: expected piece hash; context canceled; EOF
Dec 31 18:53:32.745299210 (627h24m32.616382245s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold
Dec 31 18:53:32.745319939 (627h24m32.616402985s) [seg_pos={0 0}, upload=549775]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil>
Dec 31 18:53:32.745400913 (627h24m32.616483948s) [upload=549775]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 18:53:32.745406434 (627h24m32.616489479s) [upload=549775]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 18:53:33.044217872 (627h24m32.915300907s) [upload=549775]: Done uploading segments...
Dec 31 18:53:33.044236999 (627h24m32.915320034s) [upload=549775]: Upload finished. err: <nil>
Duration: 46.461462689s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.........X..................................................] duration:
3.583339798s trace:
https://jaeger.dev.storj.io/trace/1c0a173ba9d72fc4
FAIL (2025-12-31 19:00:54 UTC)
orbiter warning: context deadline exceeded
Duration: 3.583363834s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [...XXX.X.X.X.X.X....XX.X....X...X.X...XXX..X..X........X...X] duration:
30.002572586s trace:
https://jaeger.dev.storj.io/trace/2ee1b750f361c477
FAIL (2025-12-31 19:00:52 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 31 19:00:23.101495643 (627h31m22.972578678s) [upload=549879]: Starting upload
Dec 31 19:00:23.101577880 (627h31m22.972660925s) [upload=549879]: Done starting upload
Dec 31 19:00:23.101879794 (627h31m22.97296284s) [upload=549879]: Uploading segments...
Dec 31 19:00:23.101940280 (627h31m22.973023345s) [upload=549879]: Got next segment. Inline: false
Dec 31 19:00:23.101988511 (627h31m22.973071556s) [upload=549879]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 19:00:23.101995395 (627h31m22.97307844s) [upload=549879]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 19:00:23.353914334 (627h31m23.224997369s) [seg_pos={0 0}, upload=549879]: Begin upload segment...
Dec 31 19:00:23.353974038 (627h31m23.225057083s) [seg_pos={0 0}, upload=549879]: Done begin upload segment.
Dec 31 19:00:23.354032469 (627h31m23.225115514s) [noise=true , address=45.91.116.4:4940 , piece_id=2IHA6FSU5K52JPYIATK5KWBWWM77VXJ43IOYKWHDDJ2UKSK2LUCQ, seg_pos={0 0}, upload=549879]: Uploading piece...
Dec 31 19:00:23.354067526 (627h31m23.225150571s) [noise=true , address=145.79.227.6:4257 , piece_id=3TAYNRX5YSJW4CD4UISHIG7EHMOBG6S6Z2UDUXMVUMIBMEDIR24A, seg_pos={0 0}, upload=549879]: Uploading piece...
Dec 31 19:00:23.354837121 (627h31m23.225920166s) [noise=true , address=45.91.116.13:4154 , piece_id=JLXTV7EUZ53XYZBOPDNBS7KASUDLHXLJTGGML2H2NO44NEEO6QUA, seg_pos={0 0}, upload=549879]: Uploading piece...
Dec 31 19:00:23.355369875 (627h31m23.22645292s) [noise=true , address=145.79.227.50:4375 , piece_id=ES5EAQGJVIZEAFWEZJFND6H3QOZVDQ7Y5JWCEE6S3SZZPLA7QVKQ, seg_pos={0 0}, upload=549879]: Uploading piece...
Dec 31 19:00:23.355932977 (627h31m23.227016023s) [noise=true , address=45.91.116.5:4298 , piece_id=UY5J56BLAG4KRKLQRPRVZKYPXYCOLMS5YBHWNOSPJR6NTOIQFQUQ, seg_pos={0 0}, upload=549879]: Uploading piece...
Dec 31 19:00:23.356409635 (627h31m23.22749268s) [noise=true , address=145.79.227.20:4439 , piece_id=JJL62WGMJAM6WQSTJYTNK37Q3AQ2YUBZOHA44PNTWYXRMVZGZOBA, seg_pos={0 0}, upload=549879]: Uploading piece...
Dec 31 19:00:23.356985501 (627h31m23.228068546s) [noise=true , address=145.79.227.28:4725 , piece_id=7SJLR6WNNIR5ZRXJAPE7NH7MO2LPKJWMYHAUU7QSOBGEKAXT6BOQ, seg_pos={0 0}, upload=549879]: Uploading piece...
Dec 31 19:00:23.358965412 (627h31m23.230048457s) [noise=true , address=145.79.227.38:4979 , piece_id=72R2A4B74YTPBC7OZI4UYH4XQGZC5TQSYUHEJTFJKPASIWQEBCFQ, seg_pos={0 0}, upload=549879]: Uploading piece...
Dec 31 19:00:23.359494639 (627h31m23.230577694s) [noise=true , address=77.90.31.18:4808 , piece_id=I35VLRGG7HTTPQVGWKSUAVLREWFPOFWU2Y34WXPIMDDSYND66QDA, seg_pos={0 0}, upload=549879]: Uploading piece...
Dec 31 19:00:23.360766782 (627h31m23.231849827s) [noise=true , address=145.79.227.35:4482 , piece_id=SAYSWGG4LKS4I65JR733AUNZVSJ7AY7RCSNQ4ZDRBE4LSKR3DD7A, seg_pos={0 0}, upload=549879]: Uploading piece...
Dec 31 19:00:23.362291576 (627h31m23.233374631s) [noise=true , address=145.79.227.11:4731 , piece_id=QSFELWDKMMKX5XNQQ3YCYIQINEFZACGVD5WAAJX5IAZPDGWLIKYA, seg_pos={0 0}, upload=549879]: Uploading piece...
Dec 31 19:00:23.362698179 (627h31m23.233781224s) [noise=true , address=145.79.227.5:4221 , piece_id=C22FRO5QMPT555SHKNY2J26VAT7U2UIJRO6XUBX3SN23O7QSRJ2Q, seg_pos={0 0}, upload=549879]: Uploading piece...
Dec 31 19:00:23.364507555 (627h31m23.2355906s) [noise=true , address=45.91.116.14:4190 , piece_id=BSPLFVNTI4TODWRZHXCYBINEOBBHQKNYCK7Y7YNJVQJSJSYXU5YA, seg_pos={0 0}, upload=549879]: Uploading piece...
Dec 31 19:00:23.366461655 (627h31m23.23754469s) [noise=true , address=145.79.227.10:4403 , piece_id=HVG5A5M26QDP64GF3PYDEJSWX5WLYUAS2T2MQGMFKWP2K6WCRHPQ, seg_pos={0 0}, upload=549879]: Uploading piece...
Dec 31 19:00:23.366828675 (627h31m23.23791173s) [noise=true , address=145.79.227.36:4517 , piece_id=ROTC2GJNMJHMIXMMHLNPJ52EBUK6QAN5DJUPJ5HFJAWB3LYMHTTQ, seg_pos={0 0}, upload=549879]: Uploading piece...
Dec 31 19:00:23.373516479 (627h31m23.244599524s) [noise=true , address=145.79.227.2:4114 , piece_id=CG752KK7JXUAO7QU3SQDQLT7REMT74MWFZSXGUO53FSP2SMHZFEQ, seg_pos={0 0}, upload=549879]: Uploading piece...
Dec 31 19:00:25.063752927 (627h31m24.934835962s) [upload=549879]: Next returned nil segment
Dec 31 19:00:25.063772775 (627h31m24.93485582s) [upload=549879]: Waiting for error group managing segments...
Dec 31 19:00:25.345409551 (627h31m25.216492596s) [noise=true , address=145.79.227.10:4403 , piece_id=HVG5A5M26QDP64GF3PYDEJSWX5WLYUAS2T2MQGMFKWP2K6WCRHPQ, seg_pos={0 0}, upload=549879]: Done uploading piece. err: <nil>
Dec 31 19:00:25.351796643 (627h31m25.222879678s) [noise=true , address=145.79.227.20:4439 , piece_id=JJL62WGMJAM6WQSTJYTNK37Q3AQ2YUBZOHA44PNTWYXRMVZGZOBA, seg_pos={0 0}, upload=549879]: Done uploading piece. err: <nil>
Dec 31 19:00:25.367729672 (627h31m25.238812717s) [noise=true , address=145.79.227.50:4375 , piece_id=ES5EAQGJVIZEAFWEZJFND6H3QOZVDQ7Y5JWCEE6S3SZZPLA7QVKQ, seg_pos={0 0}, upload=549879]: Done uploading piece. err: <nil>
Dec 31 19:00:25.376233765 (627h31m25.24731681s) [noise=true , address=145.79.227.11:4731 , piece_id=QSFELWDKMMKX5XNQQ3YCYIQINEFZACGVD5WAAJX5IAZPDGWLIKYA, seg_pos={0 0}, upload=549879]: Done uploading piece. err: <nil>
Dec 31 19:00:25.380237349 (627h31m25.251320384s) [noise=true , address=145.79.227.5:4221 , piece_id=C22FRO5QMPT555SHKNY2J26VAT7U2UIJRO6XUBX3SN23O7QSRJ2Q, seg_pos={0 0}, upload=549879]: Done uploading piece. err: <nil>
Dec 31 19:00:25.387752097 (627h31m25.258835142s) [noise=true , address=145.79.227.36:4517 , piece_id=ROTC2GJNMJHMIXMMHLNPJ52EBUK6QAN5DJUPJ5HFJAWB3LYMHTTQ, seg_pos={0 0}, upload=549879]: Done uploading piece. err: <nil>
Dec 31 19:00:25.391505595 (627h31m25.26258864s) [noise=true , address=145.79.227.6:4257 , piece_id=3TAYNRX5YSJW4CD4UISHIG7EHMOBG6S6Z2UDUXMVUMIBMEDIR24A, seg_pos={0 0}, upload=549879]: Done uploading piece. err: <nil>
Dec 31 19:00:25.396249588 (627h31m25.267332633s) [noise=true , address=145.79.227.28:4725 , piece_id=7SJLR6WNNIR5ZRXJAPE7NH7MO2LPKJWMYHAUU7QSOBGEKAXT6BOQ, seg_pos={0 0}, upload=549879]: Done uploading piece. err: <nil>
Dec 31 19:00:25.434938006 (627h31m25.306021061s) [noise=true , address=145.79.227.38:4979 , piece_id=72R2A4B74YTPBC7OZI4UYH4XQGZC5TQSYUHEJTFJKPASIWQEBCFQ, seg_pos={0 0}, upload=549879]: Done uploading piece. err: <nil>
Dec 31 19:00:25.455720029 (627h31m25.326803074s) [noise=true , address=145.79.227.2:4114 , piece_id=CG752KK7JXUAO7QU3SQDQLT7REMT74MWFZSXGUO53FSP2SMHZFEQ, seg_pos={0 0}, upload=549879]: Done uploading piece. err: <nil>
Dec 31 19:00:25.640436706 (627h31m25.511519741s) [noise=true , address=77.90.31.18:4808 , piece_id=I35VLRGG7HTTPQVGWKSUAVLREWFPOFWU2Y34WXPIMDDSYND66QDA, seg_pos={0 0}, upload=549879]: Done uploading piece. err: <nil>
Dec 31 19:00:25.679860344 (627h31m25.550943389s) [noise=true , address=45.91.116.13:4154 , piece_id=JLXTV7EUZ53XYZBOPDNBS7KASUDLHXLJTGGML2H2NO44NEEO6QUA, seg_pos={0 0}, upload=549879]: Done uploading piece. err: <nil>
Dec 31 19:00:25.680951972 (627h31m25.552035017s) [noise=true , address=45.91.116.5:4298 , piece_id=UY5J56BLAG4KRKLQRPRVZKYPXYCOLMS5YBHWNOSPJR6NTOIQFQUQ, seg_pos={0 0}, upload=549879]: Done uploading piece. err: <nil>
Dec 31 19:00:25.680967883 (627h31m25.552050928s) [seg_pos={0 0}, upload=549879]: Segment reached optimal threshold of 13 pieces.
Dec 31 19:00:25.681293262 (627h31m25.552376327s) [noise=true , address=45.91.116.14:4190 , piece_id=BSPLFVNTI4TODWRZHXCYBINEOBBHQKNYCK7Y7YNJVQJSJSYXU5YA, seg_pos={0 0}, upload=549879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: expected piece hash; context canceled; EOF
Dec 31 19:00:25.681303311 (627h31m25.552386356s) [noise=true , address=45.91.116.14:4190 , piece_id=BSPLFVNTI4TODWRZHXCYBINEOBBHQKNYCK7Y7YNJVQJSJSYXU5YA, seg_pos={0 0}, upload=549879]: Long tail cancellation due to optimal threshold
Dec 31 19:00:25.681311888 (627h31m25.552394943s) [noise=true , address=145.79.227.35:4482 , piece_id=SAYSWGG4LKS4I65JR733AUNZVSJ7AY7RCSNQ4ZDRBE4LSKR3DD7A, seg_pos={0 0}, upload=549879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF
Dec 31 19:00:25.681323860 (627h31m25.552406905s) [noise=true , address=145.79.227.35:4482 , piece_id=SAYSWGG4LKS4I65JR733AUNZVSJ7AY7RCSNQ4ZDRBE4LSKR3DD7A, seg_pos={0 0}, upload=549879]: Long tail cancellation due to optimal threshold
Dec 31 19:00:25.681524222 (627h31m25.552607267s) [noise=true , address=45.91.116.4:4940 , piece_id=2IHA6FSU5K52JPYIATK5KWBWWM77VXJ43IOYKWHDDJ2UKSK2LUCQ, seg_pos={0 0}, upload=549879]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: write tcp 188.166.225.170:56544->45.91.116.4:4940: use of closed network connection; write tcp 188.166.225.170:56544->45.91.116.4:4940: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:56544->45.91.116.4:4940: use of closed network connection
Dec 31 19:00:25.681543648 (627h31m25.552626694s) [noise=true , address=45.91.116.4:4940 , piece_id=2IHA6FSU5K52JPYIATK5KWBWWM77VXJ43IOYKWHDDJ2UKSK2LUCQ, seg_pos={0 0}, upload=549879]: Long tail cancellation due to optimal threshold
Dec 31 19:00:25.681576010 (627h31m25.552659055s) [seg_pos={0 0}, upload=549879]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 19:00:25.681689546 (627h31m25.552772611s) [upload=549879]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 19:00:25.681697552 (627h31m25.552780597s) [upload=549879]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 19:00:25.974972229 (627h31m25.846055274s) [upload=549879]: Done uploading segments...
Dec 31 19:00:25.974994131 (627h31m25.846077176s) [upload=549879]: Upload finished. err: <nil>
Duration: 30.002595639s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..........X.................................................] duration:
3.733755969s trace:
https://jaeger.dev.storj.io/trace/1c0a173ba9d72fc4
FAIL (2025-12-31 19:00:37 UTC)
orbiter warning: context deadline exceeded
Duration: 3.733778482s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
5.922342767s trace:
https://jaeger.dev.storj.io/trace/9459b27b9ebbd
FAIL (2025-12-31 19:00:34 UTC)
orbiter warning: context deadline exceeded
Duration: 5.922446133s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
6.311721776s trace:
https://jaeger.dev.storj.io/trace/9459b27b9ebbd
FAIL (2025-12-31 18:59:43 UTC)
orbiter warning: context deadline exceeded
Duration: 6.311749219s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
20 out of 60 is failed (threshold=3) [....XXX.X.X.X.X.X....XX.X....X...X.X...XXX..X..X........X...] duration:
6.013379937s trace:
https://jaeger.dev.storj.io/trace/1e4a8101e18af7bc
FAIL (2025-12-31 18:59:22 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: SLSO3IFJS252EG4DO7INGII2YK3UX3XZYJLEVZ2J7C6BXA3MSZRA) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: NYKFJWQ44TW6HPUJ77HVMWVI7QMDG76V7YNCJDXBPIBUPM76SRWQ) context deadline exceeded
======LOGS======
Dec 31 18:57:27.984405966 (627h28m27.855489011s) [upload=549835]: Starting upload
Dec 31 18:57:27.984517499 (627h28m27.855600534s) [upload=549835]: Done starting upload
Dec 31 18:57:27.984624893 (627h28m27.855707938s) [upload=549835]: Uploading segments...
Dec 31 18:57:27.984947226 (627h28m27.856030282s) [upload=549835]: Got next segment. Inline: false
Dec 31 18:57:27.984986212 (627h28m27.856069257s) [upload=549835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 18:57:27.984991622 (627h28m27.856074667s) [upload=549835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 18:57:28.233644711 (627h28m28.104727746s) [seg_pos={0 0}, upload=549835]: Begin upload segment...
Dec 31 18:57:28.233693825 (627h28m28.10477686s) [seg_pos={0 0}, upload=549835]: Done begin upload segment.
Dec 31 18:57:28.233735504 (627h28m28.104818539s) [noise=true , address=145.79.227.8:4331 , piece_id=TYJVVL5W2GEYDUT7FXNOMW6CQPMBJNUMOUSGGQQ6HM22VZD5HDDA, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.233774499 (627h28m28.104857534s) [noise=true , address=145.79.227.22:4509 , piece_id=RILOPLKLJ6GXRU6UOFYMS2N6HHZVRYZPP4POF4Q5ZKCFFQHMIGYA, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.234610060 (627h28m28.105693095s) [noise=true , address=45.91.116.8:4047 , piece_id=UN6C3UP3X5V2EQUVOQ2VL7F565OZQOIJ5GT2WPBVATETM42DZATQ, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.235199161 (627h28m28.106282196s) [noise=true , address=145.79.227.2:4115 , piece_id=R44YGEEC7Y6YJ5HXXAN22IIKHPJY2WHRG7G6JUFPQLFBG3WDRG2Q, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.235376108 (627h28m28.106459153s) [noise=true , address=77.90.31.11:4553 , piece_id=C2A72TC6NEDO7EPWIYV4FKCJVPYVYOTD6DAH3QM7EF5EA4TCFKLA, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.235877954 (627h28m28.106961s) [noise=true , address=77.90.31.19:4841 , piece_id=NYKFJWQ44TW6HPUJ77HVMWVI7QMDG76V7YNCJDXBPIBUPM76SRWQ, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.236300870 (627h28m28.107383905s) [noise=true , address=45.91.116.13:4153 , piece_id=ILIWDOO6BCW2AIAPA6TDXY34GENYHVW6VHBJVI4XYPFWWXGQTBVA, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.236460433 (627h28m28.107543468s) [noise=true , address=45.91.116.9:4263 , piece_id=AB5M76KTJ5ZRUG5SCWF6KQC6MDVFOQ3W46WLVCWIJ6KC5XUM3AGQ, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.236698216 (627h28m28.107781261s) [noise=true , address=45.91.116.12:4225 , piece_id=KD3X6KQDLQOGVCWDIAKIBO6KSMNU4CWMTIPJ7YUSAX5ZIE3FU2BA, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.236809628 (627h28m28.107892663s) [noise=true , address=77.90.31.16:4736 , piece_id=NUC4SR64EI27Z4QM3WJCQOOXOD3FVVZ4Y3DFPC6USGGT53Z67DAA, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.237189822 (627h28m28.108272857s) [noise=true , address=145.79.227.11:4730 , piece_id=H2MWQCECAEX3CLBX4NBWBVZKDYND5V7HOFK5E6U5LTBGHKO7T7QA, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.237221242 (627h28m28.108304277s) [noise=true , address=145.79.227.45:4231 , piece_id=WOCRM6BEA6YA4JGHNJTE25WOZDPFNBACSGZRYRBWZZKFJ2INWPNQ, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.237647082 (627h28m28.108730127s) [noise=true , address=145.79.227.21:4475 , piece_id=ABIR6LI7JBF6KDOIHPJ2DVYZZET5JVJVFPQKYWQTIQQ4DAKXGJPQ, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.238244391 (627h28m28.109327436s) [noise=true , address=145.79.227.33:4869 , piece_id=FY44KFK4VL2MOOSFNEEDZO5657KU3RTH3YOXCSMUDDW3UQYXYVIQ, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.238778848 (627h28m28.109861883s) [noise=true , address=45.91.116.6:4976 , piece_id=YDAMJRV7TKG7PLXK3PZYBEHQFNYKJNMIVQWPQWPDFLJQOVQ5QTRQ, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.239270244 (627h28m28.110353279s) [noise=true , address=45.91.116.11:4117 , piece_id=SLSO3IFJS252EG4DO7INGII2YK3UX3XZYJLEVZ2J7C6BXA3MSZRA, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:29.955700029 (627h28m29.826783064s) [upload=549835]: Next returned nil segment
Dec 31 18:57:29.955713264 (627h28m29.826796299s) [upload=549835]: Waiting for error group managing segments...
Dec 31 18:57:30.241909242 (627h28m30.112992287s) [noise=true , address=145.79.227.11:4730 , piece_id=H2MWQCECAEX3CLBX4NBWBVZKDYND5V7HOFK5E6U5LTBGHKO7T7QA, seg_pos={0 0}, upload=549835]: Done uploading piece. err: <nil>
Dec 31 18:57:30.249733430 (627h28m30.120816465s) [noise=true , address=145.79.227.45:4231 , piece_id=WOCRM6BEA6YA4JGHNJTE25WOZDPFNBACSGZRYRBWZZKFJ2INWPNQ, seg_pos={0 0}, upload=549835]: Done uploading piece. err: <nil>
Dec 31 18:57:30.251138966 (627h28m30.122222001s) [noise=true , address=145.79.227.8:4331 , piece_id=TYJVVL5W2GEYDUT7FXNOMW6CQPMBJNUMOUSGGQQ6HM22VZD5HDDA, seg_pos={0 0}, upload=549835]: Done uploading piece. err: <nil>
Dec 31 18:57:30.291763270 (627h28m30.162846305s) [noise=true , address=145.79.227.2:4115 , piece_id=R44YGEEC7Y6YJ5HXXAN22IIKHPJY2WHRG7G6JUFPQLFBG3WDRG2Q, seg_pos={0 0}, upload=549835]: Done uploading piece. err: <nil>
Dec 31 18:57:30.314325042 (627h28m30.185408077s) [noise=true , address=145.79.227.22:4509 , piece_id=RILOPLKLJ6GXRU6UOFYMS2N6HHZVRYZPP4POF4Q5ZKCFFQHMIGYA, seg_pos={0 0}, upload=549835]: Done uploading piece. err: <nil>
Dec 31 18:57:30.343534210 (627h28m30.214617255s) [noise=true , address=145.79.227.21:4475 , piece_id=ABIR6LI7JBF6KDOIHPJ2DVYZZET5JVJVFPQKYWQTIQQ4DAKXGJPQ, seg_pos={0 0}, upload=549835]: Done uploading piece. err: <nil>
Dec 31 18:57:30.446669323 (627h28m30.317752368s) [noise=true , address=77.90.31.11:4553 , piece_id=C2A72TC6NEDO7EPWIYV4FKCJVPYVYOTD6DAH3QM7EF5EA4TCFKLA, seg_pos={0 0}, upload=549835]: Done uploading piece. err: <nil>
Dec 31 18:57:30.484568208 (627h28m30.355651244s) [noise=true , address=77.90.31.16:4736 , piece_id=NUC4SR64EI27Z4QM3WJCQOOXOD3FVVZ4Y3DFPC6USGGT53Z67DAA, seg_pos={0 0}, upload=549835]: Done uploading piece. err: <nil>
Dec 31 18:57:30.537611971 (627h28m30.408695006s) [noise=true , address=77.90.31.19:4841 , piece_id=NYKFJWQ44TW6HPUJ77HVMWVI7QMDG76V7YNCJDXBPIBUPM76SRWQ, seg_pos={0 0}, upload=549835]: Done uploading piece. err: <nil>
Dec 31 18:57:30.542087543 (627h28m30.413170588s) [noise=true , address=45.91.116.12:4225 , piece_id=KD3X6KQDLQOGVCWDIAKIBO6KSMNU4CWMTIPJ7YUSAX5ZIE3FU2BA, seg_pos={0 0}, upload=549835]: Done uploading piece. err: <nil>
Dec 31 18:57:30.547628223 (627h28m30.418711258s) [noise=true , address=45.91.116.11:4117 , piece_id=SLSO3IFJS252EG4DO7INGII2YK3UX3XZYJLEVZ2J7C6BXA3MSZRA, seg_pos={0 0}, upload=549835]: Done uploading piece. err: <nil>
Dec 31 18:57:30.580679227 (627h28m30.451762263s) [noise=true , address=45.91.116.6:4976 , piece_id=YDAMJRV7TKG7PLXK3PZYBEHQFNYKJNMIVQWPQWPDFLJQOVQ5QTRQ, seg_pos={0 0}, upload=549835]: Done uploading piece. err: <nil>
Dec 31 18:57:30.584256910 (627h28m30.455339945s) [noise=true , address=45.91.116.8:4047 , piece_id=UN6C3UP3X5V2EQUVOQ2VL7F565OZQOIJ5GT2WPBVATETM42DZATQ, seg_pos={0 0}, upload=549835]: Done uploading piece. err: <nil>
Dec 31 18:57:30.584277659 (627h28m30.455360704s) [seg_pos={0 0}, upload=549835]: Segment reached optimal threshold of 13 pieces.
Dec 31 18:57:30.584506565 (627h28m30.455589611s) [noise=true , address=45.91.116.13:4153 , piece_id=ILIWDOO6BCW2AIAPA6TDXY34GENYHVW6VHBJVI4XYPFWWXGQTBVA, seg_pos={0 0}, upload=549835]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF
Dec 31 18:57:30.584514771 (627h28m30.455597816s) [noise=true , address=45.91.116.13:4153 , piece_id=ILIWDOO6BCW2AIAPA6TDXY34GENYHVW6VHBJVI4XYPFWWXGQTBVA, seg_pos={0 0}, upload=549835]: Long tail cancellation due to optimal threshold
Dec 31 18:57:30.584653645 (627h28m30.4557367s) [noise=true , address=45.91.116.9:4263 , piece_id=AB5M76KTJ5ZRUG5SCWF6KQC6MDVFOQ3W46WLVCWIJ6KC5XUM3AGQ, seg_pos={0 0}, upload=549835]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF
Dec 31 18:57:30.584664085 (627h28m30.45574712s) [noise=true , address=45.91.116.9:4263 , piece_id=AB5M76KTJ5ZRUG5SCWF6KQC6MDVFOQ3W46WLVCWIJ6KC5XUM3AGQ, seg_pos={0 0}, upload=549835]: Long tail cancellation due to optimal threshold
Dec 31 18:57:30.584682270 (627h28m30.455765315s) [noise=true , address=145.79.227.33:4869 , piece_id=FY44KFK4VL2MOOSFNEEDZO5657KU3RTH3YOXCSMUDDW3UQYXYVIQ, seg_pos={0 0}, upload=549835]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF
Dec 31 18:57:30.584689884 (627h28m30.455772939s) [noise=true , address=145.79.227.33:4869 , piece_id=FY44KFK4VL2MOOSFNEEDZO5657KU3RTH3YOXCSMUDDW3UQYXYVIQ, seg_pos={0 0}, upload=549835]: Long tail cancellation due to optimal threshold
Dec 31 18:57:30.584726304 (627h28m30.455809359s) [seg_pos={0 0}, upload=549835]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 18:57:30.584809682 (627h28m30.455892727s) [upload=549835]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 18:57:30.584817017 (627h28m30.455900062s) [upload=549835]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 18:57:30.876176376 (627h28m30.747259421s) [upload=549835]: Done uploading segments...
Dec 31 18:57:30.876197868 (627h28m30.747280903s) [upload=549835]: Upload finished. err: <nil>
Duration: 6.013403752s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...........X................................................] duration:
3.482155676s trace:
https://jaeger.dev.storj.io/trace/1c0a173ba9d72fc4
FAIL (2025-12-31 18:59:22 UTC)
orbiter warning: context deadline exceeded
Duration: 3.482182707s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [...............................X..................X.X.......] duration:
15.681708353s trace:
https://jaeger.dev.storj.io/trace/3fb4961d520cfd75
FAIL (2025-12-31 18:59:19 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12bYqoLXPqxPwKHkGRWx6RSpY8KSnS7QeKafAmtEpGJ5ThmJcDD, Piece ID: ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A) context deadline exceeded; piecestore: (Node ID: 1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, Piece ID: EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ) context deadline exceeded; piecestore: (Node ID: 1zF1SqsUdx5ikfqnG4SfeFukn8ZgX56sC7DGEPtPxZqEicws6o, Piece ID: PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA) context deadline exceeded
======LOGS======
, address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.958820533 (627h24m29.829903569s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.959307350 (627h24m29.830390386s) [noise=true , address=145.79.227.49:4349 , piece_id=R2IQZU7VSWQRVSJITAEMDNNJIPSJA3LBGC2X27AZ7GNZSN4XW2TA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.959608194 (627h24m29.830691229s) [noise=true , address=145.79.227.24:4588 , piece_id=5DP6OWR4UURMV22UA73ZIWUITQCEERVA6KE2K3T6TVDXAV2SO65A, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.959886383 (627h24m29.830969429s) [noise=true , address=64.49.13.26:4945 , piece_id=UMBXDB3DCEJERPVVINHZPPLJ7GKLDZ4PCZKBEOQD5RY5MB7LPCJQ, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.960144104 (627h24m29.831227139s) [noise=true , address=45.91.116.6:4008 , piece_id=64B6RIG5ANWZEVEMNU5HGC6X6E7EHBAIPDJN77UT6RQYYXRGVRWA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.960436030 (627h24m29.831519065s) [noise=true , address=145.79.227.47:4289 , piece_id=GMAPJV5LDECT7ZKYL23XWLDDTYHRAREVFZ2PEIH6OVJNTHUTTAHQ, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.960723447 (627h24m29.831806482s) [noise=true , address=145.79.227.42:4134 , piece_id=SWUYLD4EMMIYVLSWKSO3G3VOQ32ZGAI7NFX7KOURSIN3X6IZ4YJQ, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.960995856 (627h24m29.832078891s) [noise=true , address=145.79.227.40:4084 , piece_id=FD4PW56PCRS3QYHNXA6UQ3SXW2HNFIV3W53XFKMCQMAHDF542N3A, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.961374016 (627h24m29.832457051s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.961713552 (627h24m29.832796587s) [noise=true , address=64.49.13.137:4501 , piece_id=JBYIIB5EY4R2PJULBSLUFDQ3CMMYYIZHTS4BIX7GV5F2KY5OZC6A, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.962169091 (627h24m29.833252136s) [noise=true , address=145.79.227.2:4132 , piece_id=H2G6OZT52HEDBWWOSPU3VS5XIUT64XP5BIO2U7VAX5LNARERBQFA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.962533905 (627h24m29.83361695s) [noise=true , address=145.79.227.18:4027 , piece_id=RVLZBSJIG7UIDS6ITMGFQZJSJEKKDG4G6RFWYLXPUL7AEDZRZHMA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.963210673 (627h24m29.834293708s) [noise=true , address=145.79.227.35:4502 , piece_id=JDBTVMAGPR7M2TPMWJKMDQ562USDDYYPYFQMGEJSTNPARS6NAD5A, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.963755381 (627h24m29.834838426s) [noise=true , address=64.49.13.27:4957 , piece_id=ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.964023922 (627h24m29.835106957s) [noise=true , address=77.90.31.15:4730 , piece_id=PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.964592434 (627h24m29.835675469s) [noise=true , address=77.90.31.16:4753 , piece_id=EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.965227163 (627h24m29.836310208s) [noise=true , address=145.79.227.30:4770 , piece_id=TRAIDZYLZCU5UGHDYAJREYUELZQ5H5AFU4HHPYQFFKPXRZ5OJAQQ, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:31.410300275 (627h24m31.28138331s) [upload=549775]: Next returned nil segment
Dec 31 18:53:31.410315224 (627h24m31.281398259s) [upload=549775]: Waiting for error group managing segments...
Dec 31 18:53:31.788440578 (627h24m31.659523613s) [noise=true , address=64.49.13.26:4945 , piece_id=UMBXDB3DCEJERPVVINHZPPLJ7GKLDZ4PCZKBEOQD5RY5MB7LPCJQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.858179480 (627h24m31.729262526s) [noise=true , address=145.79.227.42:4134 , piece_id=SWUYLD4EMMIYVLSWKSO3G3VOQ32ZGAI7NFX7KOURSIN3X6IZ4YJQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.887866078 (627h24m31.758949113s) [noise=true , address=145.79.227.24:4588 , piece_id=5DP6OWR4UURMV22UA73ZIWUITQCEERVA6KE2K3T6TVDXAV2SO65A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.918774252 (627h24m31.789857297s) [noise=true , address=145.79.227.18:4027 , piece_id=RVLZBSJIG7UIDS6ITMGFQZJSJEKKDG4G6RFWYLXPUL7AEDZRZHMA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.922490298 (627h24m31.793573333s) [noise=true , address=145.79.227.35:4502 , piece_id=JDBTVMAGPR7M2TPMWJKMDQ562USDDYYPYFQMGEJSTNPARS6NAD5A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.924120612 (627h24m31.795203647s) [noise=true , address=145.79.227.2:4132 , piece_id=H2G6OZT52HEDBWWOSPU3VS5XIUT64XP5BIO2U7VAX5LNARERBQFA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.971450444 (627h24m31.842533479s) [noise=true , address=145.79.227.47:4289 , piece_id=GMAPJV5LDECT7ZKYL23XWLDDTYHRAREVFZ2PEIH6OVJNTHUTTAHQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.973634513 (627h24m31.844717548s) [noise=true , address=64.49.13.27:4957 , piece_id=ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.984147871 (627h24m31.855230906s) [noise=true , address=64.49.13.137:4501 , piece_id=JBYIIB5EY4R2PJULBSLUFDQ3CMMYYIZHTS4BIX7GV5F2KY5OZC6A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.002802704 (627h24m31.873885749s) [noise=true , address=77.90.31.15:4730 , piece_id=PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.113876665 (627h24m31.9849597s) [noise=true , address=77.90.31.16:4753 , piece_id=EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.307710471 (627h24m32.178793506s) [noise=true , address=45.91.116.6:4008 , piece_id=64B6RIG5ANWZEVEMNU5HGC6X6E7EHBAIPDJN77UT6RQYYXRGVRWA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.740289560 (627h24m32.611372605s) [noise=true , address=145.79.227.30:4770 , piece_id=TRAIDZYLZCU5UGHDYAJREYUELZQ5H5AFU4HHPYQFFKPXRZ5OJAQQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.744374660 (627h24m32.615457715s) [noise=true , address=145.79.227.49:4349 , piece_id=R2IQZU7VSWQRVSJITAEMDNNJIPSJA3LBGC2X27AZ7GNZSN4XW2TA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.744399677 (627h24m32.615482712s) [seg_pos={0 0}, upload=549775]: Segment reached optimal threshold of 14 pieces.
Dec 31 18:53:32.744624495 (627h24m32.61570753s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA): protocol: write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection; write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection
Dec 31 18:53:32.744633362 (627h24m32.615716397s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold
Dec 31 18:53:32.744938423 (627h24m32.616021468s) [noise=true , address=145.79.227.40:4084 , piece_id=FD4PW56PCRS3QYHNXA6UQ3SXW2HNFIV3W53XFKMCQMAHDF542N3A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.745025308 (627h24m32.616108343s) [noise=true , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gh7eXJMeRFyjBP2e9ZyTQVVE1mMoMFfSkQ9ssfxJyqG63C4gP): protocol: expected piece hash; context canceled; EOF
Dec 31 18:53:32.745035146 (627h24m32.616118181s) [noise=true , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold
Dec 31 18:53:32.745291255 (627h24m32.61637429s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu): protocol: expected piece hash; context canceled; EOF
Dec 31 18:53:32.745299210 (627h24m32.616382245s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold
Dec 31 18:53:32.745319939 (627h24m32.616402985s) [seg_pos={0 0}, upload=549775]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil>
Dec 31 18:53:32.745400913 (627h24m32.616483948s) [upload=549775]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 18:53:32.745406434 (627h24m32.616489479s) [upload=549775]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 18:53:33.044217872 (627h24m32.915300907s) [upload=549775]: Done uploading segments...
Dec 31 18:53:33.044236999 (627h24m32.915320034s) [upload=549775]: Upload finished. err: <nil>
Duration: 15.68172764s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
6.226299093s trace:
https://jaeger.dev.storj.io/trace/9459b27b9ebbd
FAIL (2025-12-31 18:59:14 UTC)
orbiter warning: context deadline exceeded
Duration: 6.22632441s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [................................X..................X.X......] duration:
34.859088414s trace:
https://jaeger.dev.storj.io/trace/3fb4961d520cfd75
FAIL (2025-12-31 18:59:03 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12bYqoLXPqxPwKHkGRWx6RSpY8KSnS7QeKafAmtEpGJ5ThmJcDD, Piece ID: ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A) context deadline exceeded; piecestore: (Node ID: 1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, Piece ID: EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ) context deadline exceeded; piecestore: (Node ID: 1zF1SqsUdx5ikfqnG4SfeFukn8ZgX56sC7DGEPtPxZqEicws6o, Piece ID: PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA) context deadline exceeded
======LOGS======
, address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.958820533 (627h24m29.829903569s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.959307350 (627h24m29.830390386s) [noise=true , address=145.79.227.49:4349 , piece_id=R2IQZU7VSWQRVSJITAEMDNNJIPSJA3LBGC2X27AZ7GNZSN4XW2TA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.959608194 (627h24m29.830691229s) [noise=true , address=145.79.227.24:4588 , piece_id=5DP6OWR4UURMV22UA73ZIWUITQCEERVA6KE2K3T6TVDXAV2SO65A, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.959886383 (627h24m29.830969429s) [noise=true , address=64.49.13.26:4945 , piece_id=UMBXDB3DCEJERPVVINHZPPLJ7GKLDZ4PCZKBEOQD5RY5MB7LPCJQ, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.960144104 (627h24m29.831227139s) [noise=true , address=45.91.116.6:4008 , piece_id=64B6RIG5ANWZEVEMNU5HGC6X6E7EHBAIPDJN77UT6RQYYXRGVRWA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.960436030 (627h24m29.831519065s) [noise=true , address=145.79.227.47:4289 , piece_id=GMAPJV5LDECT7ZKYL23XWLDDTYHRAREVFZ2PEIH6OVJNTHUTTAHQ, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.960723447 (627h24m29.831806482s) [noise=true , address=145.79.227.42:4134 , piece_id=SWUYLD4EMMIYVLSWKSO3G3VOQ32ZGAI7NFX7KOURSIN3X6IZ4YJQ, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.960995856 (627h24m29.832078891s) [noise=true , address=145.79.227.40:4084 , piece_id=FD4PW56PCRS3QYHNXA6UQ3SXW2HNFIV3W53XFKMCQMAHDF542N3A, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.961374016 (627h24m29.832457051s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.961713552 (627h24m29.832796587s) [noise=true , address=64.49.13.137:4501 , piece_id=JBYIIB5EY4R2PJULBSLUFDQ3CMMYYIZHTS4BIX7GV5F2KY5OZC6A, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.962169091 (627h24m29.833252136s) [noise=true , address=145.79.227.2:4132 , piece_id=H2G6OZT52HEDBWWOSPU3VS5XIUT64XP5BIO2U7VAX5LNARERBQFA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.962533905 (627h24m29.83361695s) [noise=true , address=145.79.227.18:4027 , piece_id=RVLZBSJIG7UIDS6ITMGFQZJSJEKKDG4G6RFWYLXPUL7AEDZRZHMA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.963210673 (627h24m29.834293708s) [noise=true , address=145.79.227.35:4502 , piece_id=JDBTVMAGPR7M2TPMWJKMDQ562USDDYYPYFQMGEJSTNPARS6NAD5A, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.963755381 (627h24m29.834838426s) [noise=true , address=64.49.13.27:4957 , piece_id=ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.964023922 (627h24m29.835106957s) [noise=true , address=77.90.31.15:4730 , piece_id=PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.964592434 (627h24m29.835675469s) [noise=true , address=77.90.31.16:4753 , piece_id=EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.965227163 (627h24m29.836310208s) [noise=true , address=145.79.227.30:4770 , piece_id=TRAIDZYLZCU5UGHDYAJREYUELZQ5H5AFU4HHPYQFFKPXRZ5OJAQQ, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:31.410300275 (627h24m31.28138331s) [upload=549775]: Next returned nil segment
Dec 31 18:53:31.410315224 (627h24m31.281398259s) [upload=549775]: Waiting for error group managing segments...
Dec 31 18:53:31.788440578 (627h24m31.659523613s) [noise=true , address=64.49.13.26:4945 , piece_id=UMBXDB3DCEJERPVVINHZPPLJ7GKLDZ4PCZKBEOQD5RY5MB7LPCJQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.858179480 (627h24m31.729262526s) [noise=true , address=145.79.227.42:4134 , piece_id=SWUYLD4EMMIYVLSWKSO3G3VOQ32ZGAI7NFX7KOURSIN3X6IZ4YJQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.887866078 (627h24m31.758949113s) [noise=true , address=145.79.227.24:4588 , piece_id=5DP6OWR4UURMV22UA73ZIWUITQCEERVA6KE2K3T6TVDXAV2SO65A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.918774252 (627h24m31.789857297s) [noise=true , address=145.79.227.18:4027 , piece_id=RVLZBSJIG7UIDS6ITMGFQZJSJEKKDG4G6RFWYLXPUL7AEDZRZHMA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.922490298 (627h24m31.793573333s) [noise=true , address=145.79.227.35:4502 , piece_id=JDBTVMAGPR7M2TPMWJKMDQ562USDDYYPYFQMGEJSTNPARS6NAD5A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.924120612 (627h24m31.795203647s) [noise=true , address=145.79.227.2:4132 , piece_id=H2G6OZT52HEDBWWOSPU3VS5XIUT64XP5BIO2U7VAX5LNARERBQFA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.971450444 (627h24m31.842533479s) [noise=true , address=145.79.227.47:4289 , piece_id=GMAPJV5LDECT7ZKYL23XWLDDTYHRAREVFZ2PEIH6OVJNTHUTTAHQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.973634513 (627h24m31.844717548s) [noise=true , address=64.49.13.27:4957 , piece_id=ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.984147871 (627h24m31.855230906s) [noise=true , address=64.49.13.137:4501 , piece_id=JBYIIB5EY4R2PJULBSLUFDQ3CMMYYIZHTS4BIX7GV5F2KY5OZC6A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.002802704 (627h24m31.873885749s) [noise=true , address=77.90.31.15:4730 , piece_id=PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.113876665 (627h24m31.9849597s) [noise=true , address=77.90.31.16:4753 , piece_id=EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.307710471 (627h24m32.178793506s) [noise=true , address=45.91.116.6:4008 , piece_id=64B6RIG5ANWZEVEMNU5HGC6X6E7EHBAIPDJN77UT6RQYYXRGVRWA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.740289560 (627h24m32.611372605s) [noise=true , address=145.79.227.30:4770 , piece_id=TRAIDZYLZCU5UGHDYAJREYUELZQ5H5AFU4HHPYQFFKPXRZ5OJAQQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.744374660 (627h24m32.615457715s) [noise=true , address=145.79.227.49:4349 , piece_id=R2IQZU7VSWQRVSJITAEMDNNJIPSJA3LBGC2X27AZ7GNZSN4XW2TA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.744399677 (627h24m32.615482712s) [seg_pos={0 0}, upload=549775]: Segment reached optimal threshold of 14 pieces.
Dec 31 18:53:32.744624495 (627h24m32.61570753s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA): protocol: write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection; write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection
Dec 31 18:53:32.744633362 (627h24m32.615716397s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold
Dec 31 18:53:32.744938423 (627h24m32.616021468s) [noise=true , address=145.79.227.40:4084 , piece_id=FD4PW56PCRS3QYHNXA6UQ3SXW2HNFIV3W53XFKMCQMAHDF542N3A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.745025308 (627h24m32.616108343s) [noise=true , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gh7eXJMeRFyjBP2e9ZyTQVVE1mMoMFfSkQ9ssfxJyqG63C4gP): protocol: expected piece hash; context canceled; EOF
Dec 31 18:53:32.745035146 (627h24m32.616118181s) [noise=true , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold
Dec 31 18:53:32.745291255 (627h24m32.61637429s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu): protocol: expected piece hash; context canceled; EOF
Dec 31 18:53:32.745299210 (627h24m32.616382245s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold
Dec 31 18:53:32.745319939 (627h24m32.616402985s) [seg_pos={0 0}, upload=549775]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil>
Dec 31 18:53:32.745400913 (627h24m32.616483948s) [upload=549775]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 18:53:32.745406434 (627h24m32.616489479s) [upload=549775]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 18:53:33.044217872 (627h24m32.915300907s) [upload=549775]: Done uploading segments...
Dec 31 18:53:33.044236999 (627h24m32.915320034s) [upload=549775]: Upload finished. err: <nil>
Duration: 34.859134532s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
20 out of 60 is failed (threshold=3) [.....XXX.X.X.X.X.X....XX.X....X...X.X...XXX..X..X........X..] duration:
6.325230397s trace:
https://jaeger.dev.storj.io/trace/1e4a8101e18af7bc
FAIL (2025-12-31 18:58:59 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: SLSO3IFJS252EG4DO7INGII2YK3UX3XZYJLEVZ2J7C6BXA3MSZRA) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: NYKFJWQ44TW6HPUJ77HVMWVI7QMDG76V7YNCJDXBPIBUPM76SRWQ) context deadline exceeded
======LOGS======
Dec 31 18:57:27.984405966 (627h28m27.855489011s) [upload=549835]: Starting upload
Dec 31 18:57:27.984517499 (627h28m27.855600534s) [upload=549835]: Done starting upload
Dec 31 18:57:27.984624893 (627h28m27.855707938s) [upload=549835]: Uploading segments...
Dec 31 18:57:27.984947226 (627h28m27.856030282s) [upload=549835]: Got next segment. Inline: false
Dec 31 18:57:27.984986212 (627h28m27.856069257s) [upload=549835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 18:57:27.984991622 (627h28m27.856074667s) [upload=549835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 18:57:28.233644711 (627h28m28.104727746s) [seg_pos={0 0}, upload=549835]: Begin upload segment...
Dec 31 18:57:28.233693825 (627h28m28.10477686s) [seg_pos={0 0}, upload=549835]: Done begin upload segment.
Dec 31 18:57:28.233735504 (627h28m28.104818539s) [noise=true , address=145.79.227.8:4331 , piece_id=TYJVVL5W2GEYDUT7FXNOMW6CQPMBJNUMOUSGGQQ6HM22VZD5HDDA, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.233774499 (627h28m28.104857534s) [noise=true , address=145.79.227.22:4509 , piece_id=RILOPLKLJ6GXRU6UOFYMS2N6HHZVRYZPP4POF4Q5ZKCFFQHMIGYA, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.234610060 (627h28m28.105693095s) [noise=true , address=45.91.116.8:4047 , piece_id=UN6C3UP3X5V2EQUVOQ2VL7F565OZQOIJ5GT2WPBVATETM42DZATQ, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.235199161 (627h28m28.106282196s) [noise=true , address=145.79.227.2:4115 , piece_id=R44YGEEC7Y6YJ5HXXAN22IIKHPJY2WHRG7G6JUFPQLFBG3WDRG2Q, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.235376108 (627h28m28.106459153s) [noise=true , address=77.90.31.11:4553 , piece_id=C2A72TC6NEDO7EPWIYV4FKCJVPYVYOTD6DAH3QM7EF5EA4TCFKLA, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.235877954 (627h28m28.106961s) [noise=true , address=77.90.31.19:4841 , piece_id=NYKFJWQ44TW6HPUJ77HVMWVI7QMDG76V7YNCJDXBPIBUPM76SRWQ, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.236300870 (627h28m28.107383905s) [noise=true , address=45.91.116.13:4153 , piece_id=ILIWDOO6BCW2AIAPA6TDXY34GENYHVW6VHBJVI4XYPFWWXGQTBVA, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.236460433 (627h28m28.107543468s) [noise=true , address=45.91.116.9:4263 , piece_id=AB5M76KTJ5ZRUG5SCWF6KQC6MDVFOQ3W46WLVCWIJ6KC5XUM3AGQ, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.236698216 (627h28m28.107781261s) [noise=true , address=45.91.116.12:4225 , piece_id=KD3X6KQDLQOGVCWDIAKIBO6KSMNU4CWMTIPJ7YUSAX5ZIE3FU2BA, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.236809628 (627h28m28.107892663s) [noise=true , address=77.90.31.16:4736 , piece_id=NUC4SR64EI27Z4QM3WJCQOOXOD3FVVZ4Y3DFPC6USGGT53Z67DAA, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.237189822 (627h28m28.108272857s) [noise=true , address=145.79.227.11:4730 , piece_id=H2MWQCECAEX3CLBX4NBWBVZKDYND5V7HOFK5E6U5LTBGHKO7T7QA, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.237221242 (627h28m28.108304277s) [noise=true , address=145.79.227.45:4231 , piece_id=WOCRM6BEA6YA4JGHNJTE25WOZDPFNBACSGZRYRBWZZKFJ2INWPNQ, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.237647082 (627h28m28.108730127s) [noise=true , address=145.79.227.21:4475 , piece_id=ABIR6LI7JBF6KDOIHPJ2DVYZZET5JVJVFPQKYWQTIQQ4DAKXGJPQ, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.238244391 (627h28m28.109327436s) [noise=true , address=145.79.227.33:4869 , piece_id=FY44KFK4VL2MOOSFNEEDZO5657KU3RTH3YOXCSMUDDW3UQYXYVIQ, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.238778848 (627h28m28.109861883s) [noise=true , address=45.91.116.6:4976 , piece_id=YDAMJRV7TKG7PLXK3PZYBEHQFNYKJNMIVQWPQWPDFLJQOVQ5QTRQ, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.239270244 (627h28m28.110353279s) [noise=true , address=45.91.116.11:4117 , piece_id=SLSO3IFJS252EG4DO7INGII2YK3UX3XZYJLEVZ2J7C6BXA3MSZRA, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:29.955700029 (627h28m29.826783064s) [upload=549835]: Next returned nil segment
Dec 31 18:57:29.955713264 (627h28m29.826796299s) [upload=549835]: Waiting for error group managing segments...
Dec 31 18:57:30.241909242 (627h28m30.112992287s) [noise=true , address=145.79.227.11:4730 , piece_id=H2MWQCECAEX3CLBX4NBWBVZKDYND5V7HOFK5E6U5LTBGHKO7T7QA, seg_pos={0 0}, upload=549835]: Done uploading piece. err: <nil>
Dec 31 18:57:30.249733430 (627h28m30.120816465s) [noise=true , address=145.79.227.45:4231 , piece_id=WOCRM6BEA6YA4JGHNJTE25WOZDPFNBACSGZRYRBWZZKFJ2INWPNQ, seg_pos={0 0}, upload=549835]: Done uploading piece. err: <nil>
Dec 31 18:57:30.251138966 (627h28m30.122222001s) [noise=true , address=145.79.227.8:4331 , piece_id=TYJVVL5W2GEYDUT7FXNOMW6CQPMBJNUMOUSGGQQ6HM22VZD5HDDA, seg_pos={0 0}, upload=549835]: Done uploading piece. err: <nil>
Dec 31 18:57:30.291763270 (627h28m30.162846305s) [noise=true , address=145.79.227.2:4115 , piece_id=R44YGEEC7Y6YJ5HXXAN22IIKHPJY2WHRG7G6JUFPQLFBG3WDRG2Q, seg_pos={0 0}, upload=549835]: Done uploading piece. err: <nil>
Dec 31 18:57:30.314325042 (627h28m30.185408077s) [noise=true , address=145.79.227.22:4509 , piece_id=RILOPLKLJ6GXRU6UOFYMS2N6HHZVRYZPP4POF4Q5ZKCFFQHMIGYA, seg_pos={0 0}, upload=549835]: Done uploading piece. err: <nil>
Dec 31 18:57:30.343534210 (627h28m30.214617255s) [noise=true , address=145.79.227.21:4475 , piece_id=ABIR6LI7JBF6KDOIHPJ2DVYZZET5JVJVFPQKYWQTIQQ4DAKXGJPQ, seg_pos={0 0}, upload=549835]: Done uploading piece. err: <nil>
Dec 31 18:57:30.446669323 (627h28m30.317752368s) [noise=true , address=77.90.31.11:4553 , piece_id=C2A72TC6NEDO7EPWIYV4FKCJVPYVYOTD6DAH3QM7EF5EA4TCFKLA, seg_pos={0 0}, upload=549835]: Done uploading piece. err: <nil>
Dec 31 18:57:30.484568208 (627h28m30.355651244s) [noise=true , address=77.90.31.16:4736 , piece_id=NUC4SR64EI27Z4QM3WJCQOOXOD3FVVZ4Y3DFPC6USGGT53Z67DAA, seg_pos={0 0}, upload=549835]: Done uploading piece. err: <nil>
Dec 31 18:57:30.537611971 (627h28m30.408695006s) [noise=true , address=77.90.31.19:4841 , piece_id=NYKFJWQ44TW6HPUJ77HVMWVI7QMDG76V7YNCJDXBPIBUPM76SRWQ, seg_pos={0 0}, upload=549835]: Done uploading piece. err: <nil>
Dec 31 18:57:30.542087543 (627h28m30.413170588s) [noise=true , address=45.91.116.12:4225 , piece_id=KD3X6KQDLQOGVCWDIAKIBO6KSMNU4CWMTIPJ7YUSAX5ZIE3FU2BA, seg_pos={0 0}, upload=549835]: Done uploading piece. err: <nil>
Dec 31 18:57:30.547628223 (627h28m30.418711258s) [noise=true , address=45.91.116.11:4117 , piece_id=SLSO3IFJS252EG4DO7INGII2YK3UX3XZYJLEVZ2J7C6BXA3MSZRA, seg_pos={0 0}, upload=549835]: Done uploading piece. err: <nil>
Dec 31 18:57:30.580679227 (627h28m30.451762263s) [noise=true , address=45.91.116.6:4976 , piece_id=YDAMJRV7TKG7PLXK3PZYBEHQFNYKJNMIVQWPQWPDFLJQOVQ5QTRQ, seg_pos={0 0}, upload=549835]: Done uploading piece. err: <nil>
Dec 31 18:57:30.584256910 (627h28m30.455339945s) [noise=true , address=45.91.116.8:4047 , piece_id=UN6C3UP3X5V2EQUVOQ2VL7F565OZQOIJ5GT2WPBVATETM42DZATQ, seg_pos={0 0}, upload=549835]: Done uploading piece. err: <nil>
Dec 31 18:57:30.584277659 (627h28m30.455360704s) [seg_pos={0 0}, upload=549835]: Segment reached optimal threshold of 13 pieces.
Dec 31 18:57:30.584506565 (627h28m30.455589611s) [noise=true , address=45.91.116.13:4153 , piece_id=ILIWDOO6BCW2AIAPA6TDXY34GENYHVW6VHBJVI4XYPFWWXGQTBVA, seg_pos={0 0}, upload=549835]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF
Dec 31 18:57:30.584514771 (627h28m30.455597816s) [noise=true , address=45.91.116.13:4153 , piece_id=ILIWDOO6BCW2AIAPA6TDXY34GENYHVW6VHBJVI4XYPFWWXGQTBVA, seg_pos={0 0}, upload=549835]: Long tail cancellation due to optimal threshold
Dec 31 18:57:30.584653645 (627h28m30.4557367s) [noise=true , address=45.91.116.9:4263 , piece_id=AB5M76KTJ5ZRUG5SCWF6KQC6MDVFOQ3W46WLVCWIJ6KC5XUM3AGQ, seg_pos={0 0}, upload=549835]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF
Dec 31 18:57:30.584664085 (627h28m30.45574712s) [noise=true , address=45.91.116.9:4263 , piece_id=AB5M76KTJ5ZRUG5SCWF6KQC6MDVFOQ3W46WLVCWIJ6KC5XUM3AGQ, seg_pos={0 0}, upload=549835]: Long tail cancellation due to optimal threshold
Dec 31 18:57:30.584682270 (627h28m30.455765315s) [noise=true , address=145.79.227.33:4869 , piece_id=FY44KFK4VL2MOOSFNEEDZO5657KU3RTH3YOXCSMUDDW3UQYXYVIQ, seg_pos={0 0}, upload=549835]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF
Dec 31 18:57:30.584689884 (627h28m30.455772939s) [noise=true , address=145.79.227.33:4869 , piece_id=FY44KFK4VL2MOOSFNEEDZO5657KU3RTH3YOXCSMUDDW3UQYXYVIQ, seg_pos={0 0}, upload=549835]: Long tail cancellation due to optimal threshold
Dec 31 18:57:30.584726304 (627h28m30.455809359s) [seg_pos={0 0}, upload=549835]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 18:57:30.584809682 (627h28m30.455892727s) [upload=549835]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 18:57:30.584817017 (627h28m30.455900062s) [upload=549835]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 18:57:30.876176376 (627h28m30.747259421s) [upload=549835]: Done uploading segments...
Dec 31 18:57:30.876197868 (627h28m30.747280903s) [upload=549835]: Upload finished. err: <nil>
Duration: 6.325259711s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [............X...............................................] duration:
3.564446138s trace:
https://jaeger.dev.storj.io/trace/1c0a173ba9d72fc4
FAIL (2025-12-31 18:58:55 UTC)
orbiter warning: context deadline exceeded
Duration: 3.564479462s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
5.895137862s trace:
https://jaeger.dev.storj.io/trace/9459b27b9ebbd
FAIL (2025-12-31 18:58:13 UTC)
orbiter warning: context deadline exceeded
Duration: 5.895161267s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [.................................X..................X.X.....] duration:
28.517901311s trace:
https://jaeger.dev.storj.io/trace/3fb4961d520cfd75
FAIL (2025-12-31 18:58:05 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12bYqoLXPqxPwKHkGRWx6RSpY8KSnS7QeKafAmtEpGJ5ThmJcDD, Piece ID: ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A) context deadline exceeded; piecestore: (Node ID: 1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, Piece ID: EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ) context deadline exceeded; piecestore: (Node ID: 1zF1SqsUdx5ikfqnG4SfeFukn8ZgX56sC7DGEPtPxZqEicws6o, Piece ID: PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA) context deadline exceeded
======LOGS======
, address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.958820533 (627h24m29.829903569s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.959307350 (627h24m29.830390386s) [noise=true , address=145.79.227.49:4349 , piece_id=R2IQZU7VSWQRVSJITAEMDNNJIPSJA3LBGC2X27AZ7GNZSN4XW2TA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.959608194 (627h24m29.830691229s) [noise=true , address=145.79.227.24:4588 , piece_id=5DP6OWR4UURMV22UA73ZIWUITQCEERVA6KE2K3T6TVDXAV2SO65A, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.959886383 (627h24m29.830969429s) [noise=true , address=64.49.13.26:4945 , piece_id=UMBXDB3DCEJERPVVINHZPPLJ7GKLDZ4PCZKBEOQD5RY5MB7LPCJQ, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.960144104 (627h24m29.831227139s) [noise=true , address=45.91.116.6:4008 , piece_id=64B6RIG5ANWZEVEMNU5HGC6X6E7EHBAIPDJN77UT6RQYYXRGVRWA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.960436030 (627h24m29.831519065s) [noise=true , address=145.79.227.47:4289 , piece_id=GMAPJV5LDECT7ZKYL23XWLDDTYHRAREVFZ2PEIH6OVJNTHUTTAHQ, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.960723447 (627h24m29.831806482s) [noise=true , address=145.79.227.42:4134 , piece_id=SWUYLD4EMMIYVLSWKSO3G3VOQ32ZGAI7NFX7KOURSIN3X6IZ4YJQ, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.960995856 (627h24m29.832078891s) [noise=true , address=145.79.227.40:4084 , piece_id=FD4PW56PCRS3QYHNXA6UQ3SXW2HNFIV3W53XFKMCQMAHDF542N3A, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.961374016 (627h24m29.832457051s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.961713552 (627h24m29.832796587s) [noise=true , address=64.49.13.137:4501 , piece_id=JBYIIB5EY4R2PJULBSLUFDQ3CMMYYIZHTS4BIX7GV5F2KY5OZC6A, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.962169091 (627h24m29.833252136s) [noise=true , address=145.79.227.2:4132 , piece_id=H2G6OZT52HEDBWWOSPU3VS5XIUT64XP5BIO2U7VAX5LNARERBQFA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.962533905 (627h24m29.83361695s) [noise=true , address=145.79.227.18:4027 , piece_id=RVLZBSJIG7UIDS6ITMGFQZJSJEKKDG4G6RFWYLXPUL7AEDZRZHMA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.963210673 (627h24m29.834293708s) [noise=true , address=145.79.227.35:4502 , piece_id=JDBTVMAGPR7M2TPMWJKMDQ562USDDYYPYFQMGEJSTNPARS6NAD5A, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.963755381 (627h24m29.834838426s) [noise=true , address=64.49.13.27:4957 , piece_id=ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.964023922 (627h24m29.835106957s) [noise=true , address=77.90.31.15:4730 , piece_id=PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.964592434 (627h24m29.835675469s) [noise=true , address=77.90.31.16:4753 , piece_id=EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.965227163 (627h24m29.836310208s) [noise=true , address=145.79.227.30:4770 , piece_id=TRAIDZYLZCU5UGHDYAJREYUELZQ5H5AFU4HHPYQFFKPXRZ5OJAQQ, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:31.410300275 (627h24m31.28138331s) [upload=549775]: Next returned nil segment
Dec 31 18:53:31.410315224 (627h24m31.281398259s) [upload=549775]: Waiting for error group managing segments...
Dec 31 18:53:31.788440578 (627h24m31.659523613s) [noise=true , address=64.49.13.26:4945 , piece_id=UMBXDB3DCEJERPVVINHZPPLJ7GKLDZ4PCZKBEOQD5RY5MB7LPCJQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.858179480 (627h24m31.729262526s) [noise=true , address=145.79.227.42:4134 , piece_id=SWUYLD4EMMIYVLSWKSO3G3VOQ32ZGAI7NFX7KOURSIN3X6IZ4YJQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.887866078 (627h24m31.758949113s) [noise=true , address=145.79.227.24:4588 , piece_id=5DP6OWR4UURMV22UA73ZIWUITQCEERVA6KE2K3T6TVDXAV2SO65A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.918774252 (627h24m31.789857297s) [noise=true , address=145.79.227.18:4027 , piece_id=RVLZBSJIG7UIDS6ITMGFQZJSJEKKDG4G6RFWYLXPUL7AEDZRZHMA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.922490298 (627h24m31.793573333s) [noise=true , address=145.79.227.35:4502 , piece_id=JDBTVMAGPR7M2TPMWJKMDQ562USDDYYPYFQMGEJSTNPARS6NAD5A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.924120612 (627h24m31.795203647s) [noise=true , address=145.79.227.2:4132 , piece_id=H2G6OZT52HEDBWWOSPU3VS5XIUT64XP5BIO2U7VAX5LNARERBQFA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.971450444 (627h24m31.842533479s) [noise=true , address=145.79.227.47:4289 , piece_id=GMAPJV5LDECT7ZKYL23XWLDDTYHRAREVFZ2PEIH6OVJNTHUTTAHQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.973634513 (627h24m31.844717548s) [noise=true , address=64.49.13.27:4957 , piece_id=ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.984147871 (627h24m31.855230906s) [noise=true , address=64.49.13.137:4501 , piece_id=JBYIIB5EY4R2PJULBSLUFDQ3CMMYYIZHTS4BIX7GV5F2KY5OZC6A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.002802704 (627h24m31.873885749s) [noise=true , address=77.90.31.15:4730 , piece_id=PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.113876665 (627h24m31.9849597s) [noise=true , address=77.90.31.16:4753 , piece_id=EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.307710471 (627h24m32.178793506s) [noise=true , address=45.91.116.6:4008 , piece_id=64B6RIG5ANWZEVEMNU5HGC6X6E7EHBAIPDJN77UT6RQYYXRGVRWA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.740289560 (627h24m32.611372605s) [noise=true , address=145.79.227.30:4770 , piece_id=TRAIDZYLZCU5UGHDYAJREYUELZQ5H5AFU4HHPYQFFKPXRZ5OJAQQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.744374660 (627h24m32.615457715s) [noise=true , address=145.79.227.49:4349 , piece_id=R2IQZU7VSWQRVSJITAEMDNNJIPSJA3LBGC2X27AZ7GNZSN4XW2TA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.744399677 (627h24m32.615482712s) [seg_pos={0 0}, upload=549775]: Segment reached optimal threshold of 14 pieces.
Dec 31 18:53:32.744624495 (627h24m32.61570753s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA): protocol: write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection; write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection
Dec 31 18:53:32.744633362 (627h24m32.615716397s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold
Dec 31 18:53:32.744938423 (627h24m32.616021468s) [noise=true , address=145.79.227.40:4084 , piece_id=FD4PW56PCRS3QYHNXA6UQ3SXW2HNFIV3W53XFKMCQMAHDF542N3A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.745025308 (627h24m32.616108343s) [noise=true , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gh7eXJMeRFyjBP2e9ZyTQVVE1mMoMFfSkQ9ssfxJyqG63C4gP): protocol: expected piece hash; context canceled; EOF
Dec 31 18:53:32.745035146 (627h24m32.616118181s) [noise=true , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold
Dec 31 18:53:32.745291255 (627h24m32.61637429s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu): protocol: expected piece hash; context canceled; EOF
Dec 31 18:53:32.745299210 (627h24m32.616382245s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold
Dec 31 18:53:32.745319939 (627h24m32.616402985s) [seg_pos={0 0}, upload=549775]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil>
Dec 31 18:53:32.745400913 (627h24m32.616483948s) [upload=549775]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 18:53:32.745406434 (627h24m32.616489479s) [upload=549775]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 18:53:33.044217872 (627h24m32.915300907s) [upload=549775]: Done uploading segments...
Dec 31 18:53:33.044236999 (627h24m32.915320034s) [upload=549775]: Upload finished. err: <nil>
Duration: 28.517928915s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [X.....XXX.X.X.X.X.X....XX.X....X...X.X...XXX..X..X........X.] duration:
6.076625382s trace:
https://jaeger.dev.storj.io/trace/1e4a8101e18af7bc
FAIL (2025-12-31 18:58:03 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: SLSO3IFJS252EG4DO7INGII2YK3UX3XZYJLEVZ2J7C6BXA3MSZRA) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: NYKFJWQ44TW6HPUJ77HVMWVI7QMDG76V7YNCJDXBPIBUPM76SRWQ) context deadline exceeded
======LOGS======
Dec 31 18:57:27.984405966 (627h28m27.855489011s) [upload=549835]: Starting upload
Dec 31 18:57:27.984517499 (627h28m27.855600534s) [upload=549835]: Done starting upload
Dec 31 18:57:27.984624893 (627h28m27.855707938s) [upload=549835]: Uploading segments...
Dec 31 18:57:27.984947226 (627h28m27.856030282s) [upload=549835]: Got next segment. Inline: false
Dec 31 18:57:27.984986212 (627h28m27.856069257s) [upload=549835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 18:57:27.984991622 (627h28m27.856074667s) [upload=549835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 18:57:28.233644711 (627h28m28.104727746s) [seg_pos={0 0}, upload=549835]: Begin upload segment...
Dec 31 18:57:28.233693825 (627h28m28.10477686s) [seg_pos={0 0}, upload=549835]: Done begin upload segment.
Dec 31 18:57:28.233735504 (627h28m28.104818539s) [noise=true , address=145.79.227.8:4331 , piece_id=TYJVVL5W2GEYDUT7FXNOMW6CQPMBJNUMOUSGGQQ6HM22VZD5HDDA, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.233774499 (627h28m28.104857534s) [noise=true , address=145.79.227.22:4509 , piece_id=RILOPLKLJ6GXRU6UOFYMS2N6HHZVRYZPP4POF4Q5ZKCFFQHMIGYA, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.234610060 (627h28m28.105693095s) [noise=true , address=45.91.116.8:4047 , piece_id=UN6C3UP3X5V2EQUVOQ2VL7F565OZQOIJ5GT2WPBVATETM42DZATQ, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.235199161 (627h28m28.106282196s) [noise=true , address=145.79.227.2:4115 , piece_id=R44YGEEC7Y6YJ5HXXAN22IIKHPJY2WHRG7G6JUFPQLFBG3WDRG2Q, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.235376108 (627h28m28.106459153s) [noise=true , address=77.90.31.11:4553 , piece_id=C2A72TC6NEDO7EPWIYV4FKCJVPYVYOTD6DAH3QM7EF5EA4TCFKLA, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.235877954 (627h28m28.106961s) [noise=true , address=77.90.31.19:4841 , piece_id=NYKFJWQ44TW6HPUJ77HVMWVI7QMDG76V7YNCJDXBPIBUPM76SRWQ, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.236300870 (627h28m28.107383905s) [noise=true , address=45.91.116.13:4153 , piece_id=ILIWDOO6BCW2AIAPA6TDXY34GENYHVW6VHBJVI4XYPFWWXGQTBVA, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.236460433 (627h28m28.107543468s) [noise=true , address=45.91.116.9:4263 , piece_id=AB5M76KTJ5ZRUG5SCWF6KQC6MDVFOQ3W46WLVCWIJ6KC5XUM3AGQ, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.236698216 (627h28m28.107781261s) [noise=true , address=45.91.116.12:4225 , piece_id=KD3X6KQDLQOGVCWDIAKIBO6KSMNU4CWMTIPJ7YUSAX5ZIE3FU2BA, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.236809628 (627h28m28.107892663s) [noise=true , address=77.90.31.16:4736 , piece_id=NUC4SR64EI27Z4QM3WJCQOOXOD3FVVZ4Y3DFPC6USGGT53Z67DAA, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.237189822 (627h28m28.108272857s) [noise=true , address=145.79.227.11:4730 , piece_id=H2MWQCECAEX3CLBX4NBWBVZKDYND5V7HOFK5E6U5LTBGHKO7T7QA, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.237221242 (627h28m28.108304277s) [noise=true , address=145.79.227.45:4231 , piece_id=WOCRM6BEA6YA4JGHNJTE25WOZDPFNBACSGZRYRBWZZKFJ2INWPNQ, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.237647082 (627h28m28.108730127s) [noise=true , address=145.79.227.21:4475 , piece_id=ABIR6LI7JBF6KDOIHPJ2DVYZZET5JVJVFPQKYWQTIQQ4DAKXGJPQ, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.238244391 (627h28m28.109327436s) [noise=true , address=145.79.227.33:4869 , piece_id=FY44KFK4VL2MOOSFNEEDZO5657KU3RTH3YOXCSMUDDW3UQYXYVIQ, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.238778848 (627h28m28.109861883s) [noise=true , address=45.91.116.6:4976 , piece_id=YDAMJRV7TKG7PLXK3PZYBEHQFNYKJNMIVQWPQWPDFLJQOVQ5QTRQ, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.239270244 (627h28m28.110353279s) [noise=true , address=45.91.116.11:4117 , piece_id=SLSO3IFJS252EG4DO7INGII2YK3UX3XZYJLEVZ2J7C6BXA3MSZRA, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:29.955700029 (627h28m29.826783064s) [upload=549835]: Next returned nil segment
Dec 31 18:57:29.955713264 (627h28m29.826796299s) [upload=549835]: Waiting for error group managing segments...
Dec 31 18:57:30.241909242 (627h28m30.112992287s) [noise=true , address=145.79.227.11:4730 , piece_id=H2MWQCECAEX3CLBX4NBWBVZKDYND5V7HOFK5E6U5LTBGHKO7T7QA, seg_pos={0 0}, upload=549835]: Done uploading piece. err: <nil>
Dec 31 18:57:30.249733430 (627h28m30.120816465s) [noise=true , address=145.79.227.45:4231 , piece_id=WOCRM6BEA6YA4JGHNJTE25WOZDPFNBACSGZRYRBWZZKFJ2INWPNQ, seg_pos={0 0}, upload=549835]: Done uploading piece. err: <nil>
Dec 31 18:57:30.251138966 (627h28m30.122222001s) [noise=true , address=145.79.227.8:4331 , piece_id=TYJVVL5W2GEYDUT7FXNOMW6CQPMBJNUMOUSGGQQ6HM22VZD5HDDA, seg_pos={0 0}, upload=549835]: Done uploading piece. err: <nil>
Dec 31 18:57:30.291763270 (627h28m30.162846305s) [noise=true , address=145.79.227.2:4115 , piece_id=R44YGEEC7Y6YJ5HXXAN22IIKHPJY2WHRG7G6JUFPQLFBG3WDRG2Q, seg_pos={0 0}, upload=549835]: Done uploading piece. err: <nil>
Dec 31 18:57:30.314325042 (627h28m30.185408077s) [noise=true , address=145.79.227.22:4509 , piece_id=RILOPLKLJ6GXRU6UOFYMS2N6HHZVRYZPP4POF4Q5ZKCFFQHMIGYA, seg_pos={0 0}, upload=549835]: Done uploading piece. err: <nil>
Dec 31 18:57:30.343534210 (627h28m30.214617255s) [noise=true , address=145.79.227.21:4475 , piece_id=ABIR6LI7JBF6KDOIHPJ2DVYZZET5JVJVFPQKYWQTIQQ4DAKXGJPQ, seg_pos={0 0}, upload=549835]: Done uploading piece. err: <nil>
Dec 31 18:57:30.446669323 (627h28m30.317752368s) [noise=true , address=77.90.31.11:4553 , piece_id=C2A72TC6NEDO7EPWIYV4FKCJVPYVYOTD6DAH3QM7EF5EA4TCFKLA, seg_pos={0 0}, upload=549835]: Done uploading piece. err: <nil>
Dec 31 18:57:30.484568208 (627h28m30.355651244s) [noise=true , address=77.90.31.16:4736 , piece_id=NUC4SR64EI27Z4QM3WJCQOOXOD3FVVZ4Y3DFPC6USGGT53Z67DAA, seg_pos={0 0}, upload=549835]: Done uploading piece. err: <nil>
Dec 31 18:57:30.537611971 (627h28m30.408695006s) [noise=true , address=77.90.31.19:4841 , piece_id=NYKFJWQ44TW6HPUJ77HVMWVI7QMDG76V7YNCJDXBPIBUPM76SRWQ, seg_pos={0 0}, upload=549835]: Done uploading piece. err: <nil>
Dec 31 18:57:30.542087543 (627h28m30.413170588s) [noise=true , address=45.91.116.12:4225 , piece_id=KD3X6KQDLQOGVCWDIAKIBO6KSMNU4CWMTIPJ7YUSAX5ZIE3FU2BA, seg_pos={0 0}, upload=549835]: Done uploading piece. err: <nil>
Dec 31 18:57:30.547628223 (627h28m30.418711258s) [noise=true , address=45.91.116.11:4117 , piece_id=SLSO3IFJS252EG4DO7INGII2YK3UX3XZYJLEVZ2J7C6BXA3MSZRA, seg_pos={0 0}, upload=549835]: Done uploading piece. err: <nil>
Dec 31 18:57:30.580679227 (627h28m30.451762263s) [noise=true , address=45.91.116.6:4976 , piece_id=YDAMJRV7TKG7PLXK3PZYBEHQFNYKJNMIVQWPQWPDFLJQOVQ5QTRQ, seg_pos={0 0}, upload=549835]: Done uploading piece. err: <nil>
Dec 31 18:57:30.584256910 (627h28m30.455339945s) [noise=true , address=45.91.116.8:4047 , piece_id=UN6C3UP3X5V2EQUVOQ2VL7F565OZQOIJ5GT2WPBVATETM42DZATQ, seg_pos={0 0}, upload=549835]: Done uploading piece. err: <nil>
Dec 31 18:57:30.584277659 (627h28m30.455360704s) [seg_pos={0 0}, upload=549835]: Segment reached optimal threshold of 13 pieces.
Dec 31 18:57:30.584506565 (627h28m30.455589611s) [noise=true , address=45.91.116.13:4153 , piece_id=ILIWDOO6BCW2AIAPA6TDXY34GENYHVW6VHBJVI4XYPFWWXGQTBVA, seg_pos={0 0}, upload=549835]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF
Dec 31 18:57:30.584514771 (627h28m30.455597816s) [noise=true , address=45.91.116.13:4153 , piece_id=ILIWDOO6BCW2AIAPA6TDXY34GENYHVW6VHBJVI4XYPFWWXGQTBVA, seg_pos={0 0}, upload=549835]: Long tail cancellation due to optimal threshold
Dec 31 18:57:30.584653645 (627h28m30.4557367s) [noise=true , address=45.91.116.9:4263 , piece_id=AB5M76KTJ5ZRUG5SCWF6KQC6MDVFOQ3W46WLVCWIJ6KC5XUM3AGQ, seg_pos={0 0}, upload=549835]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF
Dec 31 18:57:30.584664085 (627h28m30.45574712s) [noise=true , address=45.91.116.9:4263 , piece_id=AB5M76KTJ5ZRUG5SCWF6KQC6MDVFOQ3W46WLVCWIJ6KC5XUM3AGQ, seg_pos={0 0}, upload=549835]: Long tail cancellation due to optimal threshold
Dec 31 18:57:30.584682270 (627h28m30.455765315s) [noise=true , address=145.79.227.33:4869 , piece_id=FY44KFK4VL2MOOSFNEEDZO5657KU3RTH3YOXCSMUDDW3UQYXYVIQ, seg_pos={0 0}, upload=549835]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF
Dec 31 18:57:30.584689884 (627h28m30.455772939s) [noise=true , address=145.79.227.33:4869 , piece_id=FY44KFK4VL2MOOSFNEEDZO5657KU3RTH3YOXCSMUDDW3UQYXYVIQ, seg_pos={0 0}, upload=549835]: Long tail cancellation due to optimal threshold
Dec 31 18:57:30.584726304 (627h28m30.455809359s) [seg_pos={0 0}, upload=549835]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 18:57:30.584809682 (627h28m30.455892727s) [upload=549835]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 18:57:30.584817017 (627h28m30.455900062s) [upload=549835]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 18:57:30.876176376 (627h28m30.747259421s) [upload=549835]: Done uploading segments...
Dec 31 18:57:30.876197868 (627h28m30.747280903s) [upload=549835]: Upload finished. err: <nil>
Duration: 6.076643848s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [.X.....XXX.X.X.X.X.X....XX.X....X...X.X...XXX..X..X........X] duration:
30.003559696s trace:
https://jaeger.dev.storj.io/trace/1e4a8101e18af7bc
FAIL (2025-12-31 18:57:57 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: SLSO3IFJS252EG4DO7INGII2YK3UX3XZYJLEVZ2J7C6BXA3MSZRA) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: NYKFJWQ44TW6HPUJ77HVMWVI7QMDG76V7YNCJDXBPIBUPM76SRWQ) context deadline exceeded
======LOGS======
Dec 31 18:57:27.984405966 (627h28m27.855489011s) [upload=549835]: Starting upload
Dec 31 18:57:27.984517499 (627h28m27.855600534s) [upload=549835]: Done starting upload
Dec 31 18:57:27.984624893 (627h28m27.855707938s) [upload=549835]: Uploading segments...
Dec 31 18:57:27.984947226 (627h28m27.856030282s) [upload=549835]: Got next segment. Inline: false
Dec 31 18:57:27.984986212 (627h28m27.856069257s) [upload=549835]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 18:57:27.984991622 (627h28m27.856074667s) [upload=549835]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 18:57:28.233644711 (627h28m28.104727746s) [seg_pos={0 0}, upload=549835]: Begin upload segment...
Dec 31 18:57:28.233693825 (627h28m28.10477686s) [seg_pos={0 0}, upload=549835]: Done begin upload segment.
Dec 31 18:57:28.233735504 (627h28m28.104818539s) [noise=true , address=145.79.227.8:4331 , piece_id=TYJVVL5W2GEYDUT7FXNOMW6CQPMBJNUMOUSGGQQ6HM22VZD5HDDA, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.233774499 (627h28m28.104857534s) [noise=true , address=145.79.227.22:4509 , piece_id=RILOPLKLJ6GXRU6UOFYMS2N6HHZVRYZPP4POF4Q5ZKCFFQHMIGYA, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.234610060 (627h28m28.105693095s) [noise=true , address=45.91.116.8:4047 , piece_id=UN6C3UP3X5V2EQUVOQ2VL7F565OZQOIJ5GT2WPBVATETM42DZATQ, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.235199161 (627h28m28.106282196s) [noise=true , address=145.79.227.2:4115 , piece_id=R44YGEEC7Y6YJ5HXXAN22IIKHPJY2WHRG7G6JUFPQLFBG3WDRG2Q, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.235376108 (627h28m28.106459153s) [noise=true , address=77.90.31.11:4553 , piece_id=C2A72TC6NEDO7EPWIYV4FKCJVPYVYOTD6DAH3QM7EF5EA4TCFKLA, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.235877954 (627h28m28.106961s) [noise=true , address=77.90.31.19:4841 , piece_id=NYKFJWQ44TW6HPUJ77HVMWVI7QMDG76V7YNCJDXBPIBUPM76SRWQ, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.236300870 (627h28m28.107383905s) [noise=true , address=45.91.116.13:4153 , piece_id=ILIWDOO6BCW2AIAPA6TDXY34GENYHVW6VHBJVI4XYPFWWXGQTBVA, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.236460433 (627h28m28.107543468s) [noise=true , address=45.91.116.9:4263 , piece_id=AB5M76KTJ5ZRUG5SCWF6KQC6MDVFOQ3W46WLVCWIJ6KC5XUM3AGQ, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.236698216 (627h28m28.107781261s) [noise=true , address=45.91.116.12:4225 , piece_id=KD3X6KQDLQOGVCWDIAKIBO6KSMNU4CWMTIPJ7YUSAX5ZIE3FU2BA, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.236809628 (627h28m28.107892663s) [noise=true , address=77.90.31.16:4736 , piece_id=NUC4SR64EI27Z4QM3WJCQOOXOD3FVVZ4Y3DFPC6USGGT53Z67DAA, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.237189822 (627h28m28.108272857s) [noise=true , address=145.79.227.11:4730 , piece_id=H2MWQCECAEX3CLBX4NBWBVZKDYND5V7HOFK5E6U5LTBGHKO7T7QA, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.237221242 (627h28m28.108304277s) [noise=true , address=145.79.227.45:4231 , piece_id=WOCRM6BEA6YA4JGHNJTE25WOZDPFNBACSGZRYRBWZZKFJ2INWPNQ, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.237647082 (627h28m28.108730127s) [noise=true , address=145.79.227.21:4475 , piece_id=ABIR6LI7JBF6KDOIHPJ2DVYZZET5JVJVFPQKYWQTIQQ4DAKXGJPQ, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.238244391 (627h28m28.109327436s) [noise=true , address=145.79.227.33:4869 , piece_id=FY44KFK4VL2MOOSFNEEDZO5657KU3RTH3YOXCSMUDDW3UQYXYVIQ, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.238778848 (627h28m28.109861883s) [noise=true , address=45.91.116.6:4976 , piece_id=YDAMJRV7TKG7PLXK3PZYBEHQFNYKJNMIVQWPQWPDFLJQOVQ5QTRQ, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:28.239270244 (627h28m28.110353279s) [noise=true , address=45.91.116.11:4117 , piece_id=SLSO3IFJS252EG4DO7INGII2YK3UX3XZYJLEVZ2J7C6BXA3MSZRA, seg_pos={0 0}, upload=549835]: Uploading piece...
Dec 31 18:57:29.955700029 (627h28m29.826783064s) [upload=549835]: Next returned nil segment
Dec 31 18:57:29.955713264 (627h28m29.826796299s) [upload=549835]: Waiting for error group managing segments...
Dec 31 18:57:30.241909242 (627h28m30.112992287s) [noise=true , address=145.79.227.11:4730 , piece_id=H2MWQCECAEX3CLBX4NBWBVZKDYND5V7HOFK5E6U5LTBGHKO7T7QA, seg_pos={0 0}, upload=549835]: Done uploading piece. err: <nil>
Dec 31 18:57:30.249733430 (627h28m30.120816465s) [noise=true , address=145.79.227.45:4231 , piece_id=WOCRM6BEA6YA4JGHNJTE25WOZDPFNBACSGZRYRBWZZKFJ2INWPNQ, seg_pos={0 0}, upload=549835]: Done uploading piece. err: <nil>
Dec 31 18:57:30.251138966 (627h28m30.122222001s) [noise=true , address=145.79.227.8:4331 , piece_id=TYJVVL5W2GEYDUT7FXNOMW6CQPMBJNUMOUSGGQQ6HM22VZD5HDDA, seg_pos={0 0}, upload=549835]: Done uploading piece. err: <nil>
Dec 31 18:57:30.291763270 (627h28m30.162846305s) [noise=true , address=145.79.227.2:4115 , piece_id=R44YGEEC7Y6YJ5HXXAN22IIKHPJY2WHRG7G6JUFPQLFBG3WDRG2Q, seg_pos={0 0}, upload=549835]: Done uploading piece. err: <nil>
Dec 31 18:57:30.314325042 (627h28m30.185408077s) [noise=true , address=145.79.227.22:4509 , piece_id=RILOPLKLJ6GXRU6UOFYMS2N6HHZVRYZPP4POF4Q5ZKCFFQHMIGYA, seg_pos={0 0}, upload=549835]: Done uploading piece. err: <nil>
Dec 31 18:57:30.343534210 (627h28m30.214617255s) [noise=true , address=145.79.227.21:4475 , piece_id=ABIR6LI7JBF6KDOIHPJ2DVYZZET5JVJVFPQKYWQTIQQ4DAKXGJPQ, seg_pos={0 0}, upload=549835]: Done uploading piece. err: <nil>
Dec 31 18:57:30.446669323 (627h28m30.317752368s) [noise=true , address=77.90.31.11:4553 , piece_id=C2A72TC6NEDO7EPWIYV4FKCJVPYVYOTD6DAH3QM7EF5EA4TCFKLA, seg_pos={0 0}, upload=549835]: Done uploading piece. err: <nil>
Dec 31 18:57:30.484568208 (627h28m30.355651244s) [noise=true , address=77.90.31.16:4736 , piece_id=NUC4SR64EI27Z4QM3WJCQOOXOD3FVVZ4Y3DFPC6USGGT53Z67DAA, seg_pos={0 0}, upload=549835]: Done uploading piece. err: <nil>
Dec 31 18:57:30.537611971 (627h28m30.408695006s) [noise=true , address=77.90.31.19:4841 , piece_id=NYKFJWQ44TW6HPUJ77HVMWVI7QMDG76V7YNCJDXBPIBUPM76SRWQ, seg_pos={0 0}, upload=549835]: Done uploading piece. err: <nil>
Dec 31 18:57:30.542087543 (627h28m30.413170588s) [noise=true , address=45.91.116.12:4225 , piece_id=KD3X6KQDLQOGVCWDIAKIBO6KSMNU4CWMTIPJ7YUSAX5ZIE3FU2BA, seg_pos={0 0}, upload=549835]: Done uploading piece. err: <nil>
Dec 31 18:57:30.547628223 (627h28m30.418711258s) [noise=true , address=45.91.116.11:4117 , piece_id=SLSO3IFJS252EG4DO7INGII2YK3UX3XZYJLEVZ2J7C6BXA3MSZRA, seg_pos={0 0}, upload=549835]: Done uploading piece. err: <nil>
Dec 31 18:57:30.580679227 (627h28m30.451762263s) [noise=true , address=45.91.116.6:4976 , piece_id=YDAMJRV7TKG7PLXK3PZYBEHQFNYKJNMIVQWPQWPDFLJQOVQ5QTRQ, seg_pos={0 0}, upload=549835]: Done uploading piece. err: <nil>
Dec 31 18:57:30.584256910 (627h28m30.455339945s) [noise=true , address=45.91.116.8:4047 , piece_id=UN6C3UP3X5V2EQUVOQ2VL7F565OZQOIJ5GT2WPBVATETM42DZATQ, seg_pos={0 0}, upload=549835]: Done uploading piece. err: <nil>
Dec 31 18:57:30.584277659 (627h28m30.455360704s) [seg_pos={0 0}, upload=549835]: Segment reached optimal threshold of 13 pieces.
Dec 31 18:57:30.584506565 (627h28m30.455589611s) [noise=true , address=45.91.116.13:4153 , piece_id=ILIWDOO6BCW2AIAPA6TDXY34GENYHVW6VHBJVI4XYPFWWXGQTBVA, seg_pos={0 0}, upload=549835]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF
Dec 31 18:57:30.584514771 (627h28m30.455597816s) [noise=true , address=45.91.116.13:4153 , piece_id=ILIWDOO6BCW2AIAPA6TDXY34GENYHVW6VHBJVI4XYPFWWXGQTBVA, seg_pos={0 0}, upload=549835]: Long tail cancellation due to optimal threshold
Dec 31 18:57:30.584653645 (627h28m30.4557367s) [noise=true , address=45.91.116.9:4263 , piece_id=AB5M76KTJ5ZRUG5SCWF6KQC6MDVFOQ3W46WLVCWIJ6KC5XUM3AGQ, seg_pos={0 0}, upload=549835]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF
Dec 31 18:57:30.584664085 (627h28m30.45574712s) [noise=true , address=45.91.116.9:4263 , piece_id=AB5M76KTJ5ZRUG5SCWF6KQC6MDVFOQ3W46WLVCWIJ6KC5XUM3AGQ, seg_pos={0 0}, upload=549835]: Long tail cancellation due to optimal threshold
Dec 31 18:57:30.584682270 (627h28m30.455765315s) [noise=true , address=145.79.227.33:4869 , piece_id=FY44KFK4VL2MOOSFNEEDZO5657KU3RTH3YOXCSMUDDW3UQYXYVIQ, seg_pos={0 0}, upload=549835]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF
Dec 31 18:57:30.584689884 (627h28m30.455772939s) [noise=true , address=145.79.227.33:4869 , piece_id=FY44KFK4VL2MOOSFNEEDZO5657KU3RTH3YOXCSMUDDW3UQYXYVIQ, seg_pos={0 0}, upload=549835]: Long tail cancellation due to optimal threshold
Dec 31 18:57:30.584726304 (627h28m30.455809359s) [seg_pos={0 0}, upload=549835]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 18:57:30.584809682 (627h28m30.455892727s) [upload=549835]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 18:57:30.584817017 (627h28m30.455900062s) [upload=549835]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 18:57:30.876176376 (627h28m30.747259421s) [upload=549835]: Done uploading segments...
Dec 31 18:57:30.876197868 (627h28m30.747280903s) [upload=549835]: Upload finished. err: <nil>
Duration: 30.003584854s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.............X..............................................] duration:
3.447459349s trace:
https://jaeger.dev.storj.io/trace/1c0a173ba9d72fc4
FAIL (2025-12-31 18:57:50 UTC)
orbiter warning: context deadline exceeded
Duration: 3.447487582s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [..................................X..................X.X....] duration:
20.229944848s trace:
https://jaeger.dev.storj.io/trace/3fb4961d520cfd75
FAIL (2025-12-31 18:57:30 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12bYqoLXPqxPwKHkGRWx6RSpY8KSnS7QeKafAmtEpGJ5ThmJcDD, Piece ID: ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A) context deadline exceeded; piecestore: (Node ID: 1MUzLDSzWMc3Eru4zGrH3yF7aGiEmfGvzfUTm9cKNagjytQ7bE, Piece ID: EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ) context deadline exceeded; piecestore: (Node ID: 1zF1SqsUdx5ikfqnG4SfeFukn8ZgX56sC7DGEPtPxZqEicws6o, Piece ID: PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA) context deadline exceeded
======LOGS======
, address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.958820533 (627h24m29.829903569s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.959307350 (627h24m29.830390386s) [noise=true , address=145.79.227.49:4349 , piece_id=R2IQZU7VSWQRVSJITAEMDNNJIPSJA3LBGC2X27AZ7GNZSN4XW2TA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.959608194 (627h24m29.830691229s) [noise=true , address=145.79.227.24:4588 , piece_id=5DP6OWR4UURMV22UA73ZIWUITQCEERVA6KE2K3T6TVDXAV2SO65A, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.959886383 (627h24m29.830969429s) [noise=true , address=64.49.13.26:4945 , piece_id=UMBXDB3DCEJERPVVINHZPPLJ7GKLDZ4PCZKBEOQD5RY5MB7LPCJQ, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.960144104 (627h24m29.831227139s) [noise=true , address=45.91.116.6:4008 , piece_id=64B6RIG5ANWZEVEMNU5HGC6X6E7EHBAIPDJN77UT6RQYYXRGVRWA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.960436030 (627h24m29.831519065s) [noise=true , address=145.79.227.47:4289 , piece_id=GMAPJV5LDECT7ZKYL23XWLDDTYHRAREVFZ2PEIH6OVJNTHUTTAHQ, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.960723447 (627h24m29.831806482s) [noise=true , address=145.79.227.42:4134 , piece_id=SWUYLD4EMMIYVLSWKSO3G3VOQ32ZGAI7NFX7KOURSIN3X6IZ4YJQ, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.960995856 (627h24m29.832078891s) [noise=true , address=145.79.227.40:4084 , piece_id=FD4PW56PCRS3QYHNXA6UQ3SXW2HNFIV3W53XFKMCQMAHDF542N3A, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.961374016 (627h24m29.832457051s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.961713552 (627h24m29.832796587s) [noise=true , address=64.49.13.137:4501 , piece_id=JBYIIB5EY4R2PJULBSLUFDQ3CMMYYIZHTS4BIX7GV5F2KY5OZC6A, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.962169091 (627h24m29.833252136s) [noise=true , address=145.79.227.2:4132 , piece_id=H2G6OZT52HEDBWWOSPU3VS5XIUT64XP5BIO2U7VAX5LNARERBQFA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.962533905 (627h24m29.83361695s) [noise=true , address=145.79.227.18:4027 , piece_id=RVLZBSJIG7UIDS6ITMGFQZJSJEKKDG4G6RFWYLXPUL7AEDZRZHMA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.963210673 (627h24m29.834293708s) [noise=true , address=145.79.227.35:4502 , piece_id=JDBTVMAGPR7M2TPMWJKMDQ562USDDYYPYFQMGEJSTNPARS6NAD5A, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.963755381 (627h24m29.834838426s) [noise=true , address=64.49.13.27:4957 , piece_id=ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.964023922 (627h24m29.835106957s) [noise=true , address=77.90.31.15:4730 , piece_id=PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.964592434 (627h24m29.835675469s) [noise=true , address=77.90.31.16:4753 , piece_id=EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:29.965227163 (627h24m29.836310208s) [noise=true , address=145.79.227.30:4770 , piece_id=TRAIDZYLZCU5UGHDYAJREYUELZQ5H5AFU4HHPYQFFKPXRZ5OJAQQ, seg_pos={0 0}, upload=549775]: Uploading piece...
Dec 31 18:53:31.410300275 (627h24m31.28138331s) [upload=549775]: Next returned nil segment
Dec 31 18:53:31.410315224 (627h24m31.281398259s) [upload=549775]: Waiting for error group managing segments...
Dec 31 18:53:31.788440578 (627h24m31.659523613s) [noise=true , address=64.49.13.26:4945 , piece_id=UMBXDB3DCEJERPVVINHZPPLJ7GKLDZ4PCZKBEOQD5RY5MB7LPCJQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.858179480 (627h24m31.729262526s) [noise=true , address=145.79.227.42:4134 , piece_id=SWUYLD4EMMIYVLSWKSO3G3VOQ32ZGAI7NFX7KOURSIN3X6IZ4YJQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.887866078 (627h24m31.758949113s) [noise=true , address=145.79.227.24:4588 , piece_id=5DP6OWR4UURMV22UA73ZIWUITQCEERVA6KE2K3T6TVDXAV2SO65A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.918774252 (627h24m31.789857297s) [noise=true , address=145.79.227.18:4027 , piece_id=RVLZBSJIG7UIDS6ITMGFQZJSJEKKDG4G6RFWYLXPUL7AEDZRZHMA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.922490298 (627h24m31.793573333s) [noise=true , address=145.79.227.35:4502 , piece_id=JDBTVMAGPR7M2TPMWJKMDQ562USDDYYPYFQMGEJSTNPARS6NAD5A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.924120612 (627h24m31.795203647s) [noise=true , address=145.79.227.2:4132 , piece_id=H2G6OZT52HEDBWWOSPU3VS5XIUT64XP5BIO2U7VAX5LNARERBQFA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.971450444 (627h24m31.842533479s) [noise=true , address=145.79.227.47:4289 , piece_id=GMAPJV5LDECT7ZKYL23XWLDDTYHRAREVFZ2PEIH6OVJNTHUTTAHQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.973634513 (627h24m31.844717548s) [noise=true , address=64.49.13.27:4957 , piece_id=ZX7CUJMNPGMXJ4TVWVAMZY6DBXVVY6RE4PDP2V63QSSRHFRGSW5A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:31.984147871 (627h24m31.855230906s) [noise=true , address=64.49.13.137:4501 , piece_id=JBYIIB5EY4R2PJULBSLUFDQ3CMMYYIZHTS4BIX7GV5F2KY5OZC6A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.002802704 (627h24m31.873885749s) [noise=true , address=77.90.31.15:4730 , piece_id=PTBDKPNBHLSGZZC3LZLCCN7MYBB5JRGAZG7HIDVDIBDXEWP63MDA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.113876665 (627h24m31.9849597s) [noise=true , address=77.90.31.16:4753 , piece_id=EQO33KYSPJWSUNELYWD5S3TTHFWWQU3J35SSE4CJC35TRMJP22GQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.307710471 (627h24m32.178793506s) [noise=true , address=45.91.116.6:4008 , piece_id=64B6RIG5ANWZEVEMNU5HGC6X6E7EHBAIPDJN77UT6RQYYXRGVRWA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.740289560 (627h24m32.611372605s) [noise=true , address=145.79.227.30:4770 , piece_id=TRAIDZYLZCU5UGHDYAJREYUELZQ5H5AFU4HHPYQFFKPXRZ5OJAQQ, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.744374660 (627h24m32.615457715s) [noise=true , address=145.79.227.49:4349 , piece_id=R2IQZU7VSWQRVSJITAEMDNNJIPSJA3LBGC2X27AZ7GNZSN4XW2TA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.744399677 (627h24m32.615482712s) [seg_pos={0 0}, upload=549775]: Segment reached optimal threshold of 14 pieces.
Dec 31 18:53:32.744624495 (627h24m32.61570753s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1S9k5eejJcxkiG5pXjjBZm18uP8k5jCGfWyeW2QQxmuRU8ZzRA): protocol: write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection; write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58572->145.79.227.19:4087: use of closed network connection
Dec 31 18:53:32.744633362 (627h24m32.615716397s) [noise=true , address=145.79.227.19:4087 , piece_id=UFMBJYL52ICAWYB7BDNP4FLYTDJLTIJVLI6D62W76LFMQQUWFE4Q, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold
Dec 31 18:53:32.744938423 (627h24m32.616021468s) [noise=true , address=145.79.227.40:4084 , piece_id=FD4PW56PCRS3QYHNXA6UQ3SXW2HNFIV3W53XFKMCQMAHDF542N3A, seg_pos={0 0}, upload=549775]: Done uploading piece. err: <nil>
Dec 31 18:53:32.745025308 (627h24m32.616108343s) [noise=true , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gh7eXJMeRFyjBP2e9ZyTQVVE1mMoMFfSkQ9ssfxJyqG63C4gP): protocol: expected piece hash; context canceled; EOF
Dec 31 18:53:32.745035146 (627h24m32.616118181s) [noise=true , address=145.79.227.43:4179 , piece_id=TW27IRUGRYVNJXY7ELXW3PPU3SFQZWHBHW6NVGGMBNAYA7EUJHDA, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold
Dec 31 18:53:32.745291255 (627h24m32.61637429s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pCgoSfxjtXPw5jpbz9RJaxRiGoYPdPFdoMPtD19F1ZWY8UwPu): protocol: expected piece hash; context canceled; EOF
Dec 31 18:53:32.745299210 (627h24m32.616382245s) [noise=true , address=145.79.227.34:4931 , piece_id=TKXKQGTZTK6HX5PRMUPIIZ2CBPTRV2VQUOILIX4WQZOFE4CAR7NA, seg_pos={0 0}, upload=549775]: Long tail cancellation due to optimal threshold
Dec 31 18:53:32.745319939 (627h24m32.616402985s) [seg_pos={0 0}, upload=549775]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil>
Dec 31 18:53:32.745400913 (627h24m32.616483948s) [upload=549775]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 18:53:32.745406434 (627h24m32.616489479s) [upload=549775]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 18:53:33.044217872 (627h24m32.915300907s) [upload=549775]: Done uploading segments...
Dec 31 18:53:33.044236999 (627h24m32.915320034s) [upload=549775]: Upload finished. err: <nil>
Duration: 20.229974054s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
6.047074253s trace:
https://jaeger.dev.storj.io/trace/9459b27b9ebbd
FAIL (2025-12-31 18:57:15 UTC)
orbiter warning: context deadline exceeded
Duration: 6.047097958s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [X.X.....XXX.X.X.X.X.X....XX.X....X...X.X...XXX..X..X........] duration:
5.76637071s trace:
https://jaeger.dev.storj.io/trace/2f86366cb6f77d8e
FAIL (2025-12-31 18:56:52 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: 3UZJJFUBSUQO7NFIVXHULOLQJ4JGKWTDTLIQ7GZQMBDMERNESMIQ) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: BY4WPLFRDGZUX6NQM3QY3QXRBKXZCGZQGJAENXWTCNCK2X7K6VXQ) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: Z3WZWWK2JBIWTRPJUCJU4A43DRAH4MC42H6CNQCX6PAJ66JZBUBQ) context deadline exceeded
======LOGS======
Dec 31 18:51:17.512545337 (627h22m17.383628372s) [upload=549740]: Starting upload
Dec 31 18:51:17.512637953 (627h22m17.383720998s) [upload=549740]: Done starting upload
Dec 31 18:51:17.512738755 (627h22m17.3838218s) [upload=549740]: Uploading segments...
Dec 31 18:51:17.512939187 (627h22m17.384022222s) [upload=549740]: Got next segment. Inline: false
Dec 31 18:51:17.512982058 (627h22m17.384065093s) [upload=549740]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 31 18:51:17.512987448 (627h22m17.384070483s) [upload=549740]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 31 18:51:17.761956390 (627h22m17.633039435s) [seg_pos={0 0}, upload=549740]: Begin upload segment...
Dec 31 18:51:17.762015212 (627h22m17.633098257s) [seg_pos={0 0}, upload=549740]: Done begin upload segment.
Dec 31 18:51:17.762378534 (627h22m17.633461589s) [noise=true , address=45.91.116.10:4084 , piece_id=BY4WPLFRDGZUX6NQM3QY3QXRBKXZCGZQGJAENXWTCNCK2X7K6VXQ, seg_pos={0 0}, upload=549740]: Uploading piece...
Dec 31 18:51:17.763733514 (627h22m17.634816549s) [noise=true , address=145.79.227.11:4731 , piece_id=PVZP4HLS2DXF3PWAPO4OTR6N6QI3E7JFOVXZ37DKLV4YRB5QGRXQ, seg_pos={0 0}, upload=549740]: Uploading piece...
Dec 31 18:51:17.764634750 (627h22m17.635717785s) [noise=true , address=145.79.227.36:4518 , piece_id=NAKX7KJOWM44VBBGWNKKH2ZQEXR2G4BKT6GGT3RPRM53O4UWIT3A, seg_pos={0 0}, upload=549740]: Uploading piece...
Dec 31 18:51:17.765665172 (627h22m17.636748207s) [noise=true , address=145.79.227.19:4077 , piece_id=3C5RIE7VRY3K2BGE326XO5C75ROLCXHJOBFHETTNBYTBHGGPHVDA, seg_pos={0 0}, upload=549740]: Uploading piece...
Dec 31 18:51:17.766969565 (627h22m17.63805261s) [noise=true , address=77.90.31.16:4735 , piece_id=K3QN6JAYIB4GZKGZNU3AFTCJNG7ISXGO5T2WISLUVS6UODQ5VC4A, seg_pos={0 0}, upload=549740]: Uploading piece...
Dec 31 18:51:17.767390808 (627h22m17.638473843s) [noise=true , address=145.79.227.51:4410 , piece_id=S4HFGNTZHBUIGGSKHE2H5UGN5574IMRS5HUDACLJQD7HQ63ZQL7Q, seg_pos={0 0}, upload=549740]: Uploading piece...
Dec 31 18:51:17.767714754 (627h22m17.6387978s) [noise=true , address=77.90.31.14:4663 , piece_id=EZ3DQAFBEMYOMFSBYJDFJ7DO5KSGOCO4ZLHWYN6EMAKIJAYQVF5Q, seg_pos={0 0}, upload=549740]: Uploading piece...
Dec 31 18:51:17.768549203 (627h22m17.639632249s) [noise=true , address=45.91.116.4:4939 , piece_id=573B2GPT2CUKNDQVQMQVWRJC7E66MEWOPDUCLBPUT6XT7UMTGOYQ, seg_pos={0 0}, upload=549740]: Uploading piece...
Dec 31 18:51:17.769939160 (627h22m17.641022205s) [noise=true , address=145.79.227.42:4121 , piece_id=DYPG3IDJIBQ2MBM3RPIE4E7DAJDZXIQRKMSE2TMW3II5U4PRMXEA, seg_pos={0 0}, upload=549740]: Uploading piece...
Dec 31 18:51:17.770540024 (627h22m17.641623059s) [noise=true , address=77.90.31.12:4590 , piece_id=Z3WZWWK2JBIWTRPJUCJU4A43DRAH4MC42H6CNQCX6PAJ66JZBUBQ, seg_pos={0 0}, upload=549740]: Uploading piece...
Dec 31 18:51:17.770926891 (627h22m17.642009926s) [noise=true , address=45.91.116.14:4191 , piece_id=CNQ3U2JMQ6GUS6HQF745CGASHWJWKBB4P4DANLG3YX52P33VEFXA, seg_pos={0 0}, upload=549740]: Uploading piece...
Dec 31 18:51:17.771710744 (627h22m17.642793779s) [noise=true , address=145.79.227.2:4115 , piece_id=KBS52YK62X72HY6EPZ4UUPHK75OR6JRZCEJCWKX2O7JTOA5X474A, seg_pos={0 0}, upload=549740]: Uploading piece...
Dec 31 18:51:17.773200480 (627h22m17.644283525s) [noise=true , address=145.79.227.4:4185 , piece_id=LUSS4BIWP7DM377X5UH7OLNDOY22S64NXWSYOPYRN5GCWDVSVWPA, seg_pos={0 0}, upload=549740]: Uploading piece...
Dec 31 18:51:17.773478429 (627h22m17.644561474s) [noise=true , address=145.79.227.32:4834 , piece_id=6X7QWYKFOUZXB4ROBBSMHIHJYWJF6Q7JY475PPTY3CDXGMVR2NIQ, seg_pos={0 0}, upload=549740]: Uploading piece...
Dec 31 18:51:17.773884142 (627h22m17.644967177s) [noise=true , address=45.91.116.7:4012 , piece_id=GHP65MP56NN4KMNFUOGRDOGLKQEQZXDTSXXM6ZPG7D7UHQ5VGP5Q, seg_pos={0 0}, upload=549740]: Uploading piece...
Dec 31 18:51:17.777234892 (627h22m17.648317937s) [noise=true , address=77.90.31.18:4805 , piece_id=3UZJJFUBSUQO7NFIVXHULOLQJ4JGKWTDTLIQ7GZQMBDMERNESMIQ, seg_pos={0 0}, upload=549740]: Uploading piece...
Dec 31 18:51:19.474161832 (627h22m19.345244867s) [upload=549740]: Next returned nil segment
Dec 31 18:51:19.474175608 (627h22m19.345258653s) [upload=549740]: Waiting for error group managing segments...
Dec 31 18:51:19.757541637 (627h22m19.628624672s) [noise=true , address=145.79.227.19:4077 , piece_id=3C5RIE7VRY3K2BGE326XO5C75ROLCXHJOBFHETTNBYTBHGGPHVDA, seg_pos={0 0}, upload=549740]: Done uploading piece. err: <nil>
Dec 31 18:51:19.759284305 (627h22m19.630367341s) [noise=true , address=145.79.227.4:4185 , piece_id=LUSS4BIWP7DM377X5UH7OLNDOY22S64NXWSYOPYRN5GCWDVSVWPA, seg_pos={0 0}, upload=549740]: Done uploading piece. err: <nil>
Dec 31 18:51:19.774277565 (627h22m19.6453606s) [noise=true , address=145.79.227.36:4518 , piece_id=NAKX7KJOWM44VBBGWNKKH2ZQEXR2G4BKT6GGT3RPRM53O4UWIT3A, seg_pos={0 0}, upload=549740]: Done uploading piece. err: <nil>
Dec 31 18:51:19.778454638 (627h22m19.649537724s) [noise=true , address=145.79.227.2:4115 , piece_id=KBS52YK62X72HY6EPZ4UUPHK75OR6JRZCEJCWKX2O7JTOA5X474A, seg_pos={0 0}, upload=549740]: Done uploading piece. err: <nil>
Dec 31 18:51:19.780861582 (627h22m19.651944617s) [noise=true , address=145.79.227.51:4410 , piece_id=S4HFGNTZHBUIGGSKHE2H5UGN5574IMRS5HUDACLJQD7HQ63ZQL7Q, seg_pos={0 0}, upload=549740]: Done uploading piece. err: <nil>
Dec 31 18:51:19.791300688 (627h22m19.662383724s) [noise=true , address=145.79.227.42:4121 , piece_id=DYPG3IDJIBQ2MBM3RPIE4E7DAJDZXIQRKMSE2TMW3II5U4PRMXEA, seg_pos={0 0}, upload=549740]: Done uploading piece. err: <nil>
Dec 31 18:51:19.859129536 (627h22m19.730212581s) [noise=true , address=145.79.227.32:4834 , piece_id=6X7QWYKFOUZXB4ROBBSMHIHJYWJF6Q7JY475PPTY3CDXGMVR2NIQ, seg_pos={0 0}, upload=549740]: Done uploading piece. err: <nil>
Dec 31 18:51:19.916594266 (627h22m19.787677301s) [noise=true , address=77.90.31.16:4735 , piece_id=K3QN6JAYIB4GZKGZNU3AFTCJNG7ISXGO5T2WISLUVS6UODQ5VC4A, seg_pos={0 0}, upload=549740]: Done uploading piece. err: <nil>
Dec 31 18:51:19.923139279 (627h22m19.794222314s) [noise=true , address=77.90.31.14:4663 , piece_id=EZ3DQAFBEMYOMFSBYJDFJ7DO5KSGOCO4ZLHWYN6EMAKIJAYQVF5Q, seg_pos={0 0}, upload=549740]: Done uploading piece. err: <nil>
Dec 31 18:51:19.936461486 (627h22m19.807544521s) [noise=true , address=77.90.31.12:4590 , piece_id=Z3WZWWK2JBIWTRPJUCJU4A43DRAH4MC42H6CNQCX6PAJ66JZBUBQ, seg_pos={0 0}, upload=549740]: Done uploading piece. err: <nil>
Dec 31 18:51:19.945967757 (627h22m19.817050812s) [noise=true , address=77.90.31.18:4805 , piece_id=3UZJJFUBSUQO7NFIVXHULOLQJ4JGKWTDTLIQ7GZQMBDMERNESMIQ, seg_pos={0 0}, upload=549740]: Done uploading piece. err: <nil>
Dec 31 18:51:19.948722172 (627h22m19.819805217s) [noise=true , address=145.79.227.11:4731 , piece_id=PVZP4HLS2DXF3PWAPO4OTR6N6QI3E7JFOVXZ37DKLV4YRB5QGRXQ, seg_pos={0 0}, upload=549740]: Done uploading piece. err: <nil>
Dec 31 18:51:20.050306934 (627h22m19.921389969s) [noise=true , address=45.91.116.10:4084 , piece_id=BY4WPLFRDGZUX6NQM3QY3QXRBKXZCGZQGJAENXWTCNCK2X7K6VXQ, seg_pos={0 0}, upload=549740]: Done uploading piece. err: <nil>
Dec 31 18:51:20.050330379 (627h22m19.921413434s) [seg_pos={0 0}, upload=549740]: Segment reached optimal threshold of 13 pieces.
Dec 31 18:51:20.050590054 (627h22m19.921673099s) [noise=true , address=45.91.116.4:4939 , piece_id=573B2GPT2CUKNDQVQMQVWRJC7E66MEWOPDUCLBPUT6XT7UMTGOYQ, seg_pos={0 0}, upload=549740]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF
Dec 31 18:51:20.050601495 (627h22m19.92168455s) [noise=true , address=45.91.116.4:4939 , piece_id=573B2GPT2CUKNDQVQMQVWRJC7E66MEWOPDUCLBPUT6XT7UMTGOYQ, seg_pos={0 0}, upload=549740]: Long tail cancellation due to optimal threshold
Dec 31 18:51:20.050666860 (627h22m19.921749925s) [noise=true , address=45.91.116.7:4012 , piece_id=GHP65MP56NN4KMNFUOGRDOGLKQEQZXDTSXXM6ZPG7D7UHQ5VGP5Q, seg_pos={0 0}, upload=549740]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF
Dec 31 18:51:20.050675766 (627h22m19.921758801s) [noise=true , address=45.91.116.7:4012 , piece_id=GHP65MP56NN4KMNFUOGRDOGLKQEQZXDTSXXM6ZPG7D7UHQ5VGP5Q, seg_pos={0 0}, upload=549740]: Long tail cancellation due to optimal threshold
Dec 31 18:51:20.050859085 (627h22m19.92194213s) [noise=true , address=45.91.116.14:4191 , piece_id=CNQ3U2JMQ6GUS6HQF745CGASHWJWKBB4P4DANLG3YX52P33VEFXA, seg_pos={0 0}, upload=549740]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF
Dec 31 18:51:20.050868464 (627h22m19.921951499s) [noise=true , address=45.91.116.14:4191 , piece_id=CNQ3U2JMQ6GUS6HQF745CGASHWJWKBB4P4DANLG3YX52P33VEFXA, seg_pos={0 0}, upload=549740]: Long tail cancellation due to optimal threshold
Dec 31 18:51:20.050893631 (627h22m19.921976666s) [seg_pos={0 0}, upload=549740]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 31 18:51:20.050999182 (627h22m19.922082217s) [upload=549740]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 31 18:51:20.051007077 (627h22m19.922090112s) [upload=549740]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 31 18:51:20.351538912 (627h22m20.222621947s) [upload=549740]: Done uploading segments...
Dec 31 18:51:20.351557628 (627h22m20.222640673s) [upload=549740]: Upload finished. err: <nil>
Duration: 5.766403274s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
6.209326459s trace:
https://jaeger.dev.storj.io/trace/9459b27b9ebbd
FAIL (2025-12-31 18:56:43 UTC)
orbiter warning: context deadline exceeded
Duration: 6.209352839s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..............X.............................................] duration:
3.504308348s trace:
https://jaeger.dev.storj.io/trace/1c0a173ba9d72fc4
FAIL (2025-12-31 18:56:42 UTC)
orbiter warning: context deadline exceeded
Duration: 3.504338075s
core
uplink
us-select-1
us1-placement-20
|