Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
2 out of 60 is failed (threshold=3) [.......................................................X...X] duration:
1m0.000570994s trace:
https://jaeger.dev.storj.io/trace/6ea66b1ed6f4c6e5
FAIL (2025-12-30 17:10:34 UTC)
orbiter warning: context deadline exceeded
Duration: 1m0.000590401s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [......X.............X.......X...............................] duration:
29.668824393s trace:
https://jaeger.dev.storj.io/trace/3f4379d3637c8e4b
FAIL (2025-12-30 17:10:22 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1WBuBdGxdPXi6VsNf4FJpn3LpWoQztM7CV6YjAW1mUDVjAUjn2, Piece ID: MS4HNXUQIITLBMEHZDPJMEH7FSLMBBGKJSIP5BY2BDUF3IE2NKIQ) context deadline exceeded; piecestore: (Node ID: 1soiTQsiMNe5BkNHwwAGykmtP9LtLxncqE7Tv728ak8pjFeshg, Piece ID: BHVQNSBU3LP35Z5WQYTRPJZNNWTDZKWYK2CMRLHSOGBJ4IXGZ5SA) context deadline exceeded; piecestore: (Node ID: 1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, Piece ID: MVRLWV5URMPV57R52J426SXY5ZWQGHOTL42GO2GPEZ4LA7B2JGCQ) context deadline exceeded
======LOGS======
5m56.099625936s) [seg_pos={0 0}, upload=526987]: Begin upload segment...
Dec 30 16:44:56.228605921 (601h15m56.099688967s) [seg_pos={0 0}, upload=526987]: Done begin upload segment.
Dec 30 16:44:56.228736641 (601h15m56.099819686s) [noise=true , address=145.79.227.34:4923 , piece_id=OB3KQEHYXE4BL73EDDQDN2LSVYJYYQQFUSFJ655BFJISLBGOH4XQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.228753061 (601h15m56.099836117s) [noise=true , address=64.49.13.26:4947 , piece_id=BHVQNSBU3LP35Z5WQYTRPJZNNWTDZKWYK2CMRLHSOGBJ4IXGZ5SA, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.229701116 (601h15m56.100784151s) [noise=true , address=45.91.116.8:4071 , piece_id=3PUQCDJAXSQGEM47BTD2B5HWWNK3PUS6AEVQ63BEC2Q2CKCB4PXQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.230399837 (601h15m56.101482882s) [noise=true , address=77.90.31.21:4899 , piece_id=DCYPDKTC4S5LG3XEBQWZT3HGFJ63T3UC3PFUDLF7OOJHNMYW4KRQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.230834926 (601h15m56.101917971s) [noise=true , address=77.90.31.19:4853 , piece_id=MS4HNXUQIITLBMEHZDPJMEH7FSLMBBGKJSIP5BY2BDUF3IE2NKIQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.232511007 (601h15m56.103594112s) [noise=true , address=145.79.227.8:4344 , piece_id=6L5DD7CQJ5JZ7IPFX2IGHNEP7M6BZU2DA5Z5GEOKGPAMVZUXQHOQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.233047428 (601h15m56.104130473s) [noise=true , address=145.79.227.31:4808 , piece_id=MW4UFSJ4XFLXK7ZLWZFWTP2N4QQUS24WP3O44DJWT7AK2WCWAT3Q, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.233557500 (601h15m56.104640545s) [noise=true , address=145.79.227.33:4903 , piece_id=THSFTSO2Y4QVWIGNUOU4AKFGTSQWQ6FLNIIDISIVNRF7XZZWWPCQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.234065937 (601h15m56.105148983s) [noise=true , address=64.49.13.137:4493 , piece_id=QZSPIHMAGKSHSAJYJ6BR26EMTCWFAEPOPC4O65YN6SL6222H4HWQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.234513860 (601h15m56.105596896s) [noise=true , address=64.49.13.27:4965 , piece_id=AQTJD7BXKPMIAGEOGJCY5Y64IVU2SHCZEFJL2NLQAHVX5HR33N7Q, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.236125519 (601h15m56.107208554s) [noise=true , address=77.90.31.11:4567 , piece_id=NMCHY46TZ6W6EPVLVVCGQXHFERHBLSFB3J55AD32KQSKOVXABO7A, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.237179756 (601h15m56.108262791s) [noise=true , address=145.79.227.40:4064 , piece_id=SEYSDBZHX6SHMFZBGY3XDIYEOTWFCNGG2KNNTCIXYQRZP7FZOVJQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.238439144 (601h15m56.109522189s) [noise=true , address=145.79.227.30:4790 , piece_id=5VRI4FYK7J54TCOY7DX6TCQ3634N3TATGDZUZDWRBBLMXCGRVR3Q, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.239316585 (601h15m56.11039962s) [noise=true , address=145.79.227.44:4212 , piece_id=ZV3XMTVGJR2D5X4EEGGDTGGGXSQI4QNCJZCI3GPR6RTD7JEQWLGA, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.242138468 (601h15m56.113221503s) [noise=true , address=45.91.116.11:4131 , piece_id=YAFIDLBM762GWTNKDW62FMNLE4YHCNYW2WU3R4AMO5NELMVOHHHQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.243965547 (601h15m56.115048582s) [noise=true , address=77.90.31.18:4829 , piece_id=MVRLWV5URMPV57R52J426SXY5ZWQGHOTL42GO2GPEZ4LA7B2JGCQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.250805150 (601h15m56.121888185s) [noise=true , address=5.231.73.4:4033 , piece_id=CCH6AH47YBQPGPF4MPZU2NPT7HPF4D3KT3ZVDITRYF7UW5JUKWDQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.254973396 (601h15m56.126056431s) [noise=true , address=145.79.227.49:4341 , piece_id=6GKQQIBOECLDEEDZNO5AYKX7XMXZYUA6WNGKD3FQSF2Q4O6A4MQA, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:57.691193696 (601h15m57.562276731s) [upload=526987]: Next returned nil segment
Dec 30 16:44:57.691206460 (601h15m57.562289495s) [upload=526987]: Waiting for error group managing segments...
Dec 30 16:44:58.085111563 (601h15m57.956194598s) [noise=true , address=64.49.13.137:4493 , piece_id=QZSPIHMAGKSHSAJYJ6BR26EMTCWFAEPOPC4O65YN6SL6222H4HWQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.086614434 (601h15m57.957697479s) [noise=true , address=64.49.13.27:4965 , piece_id=AQTJD7BXKPMIAGEOGJCY5Y64IVU2SHCZEFJL2NLQAHVX5HR33N7Q, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.119421281 (601h15m57.990504316s) [noise=true , address=145.79.227.8:4344 , piece_id=6L5DD7CQJ5JZ7IPFX2IGHNEP7M6BZU2DA5Z5GEOKGPAMVZUXQHOQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.125317667 (601h15m57.996400702s) [noise=true , address=145.79.227.31:4808 , piece_id=MW4UFSJ4XFLXK7ZLWZFWTP2N4QQUS24WP3O44DJWT7AK2WCWAT3Q, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.262371457 (601h15m58.133454502s) [noise=true , address=64.49.13.26:4947 , piece_id=BHVQNSBU3LP35Z5WQYTRPJZNNWTDZKWYK2CMRLHSOGBJ4IXGZ5SA, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.284980216 (601h15m58.156063251s) [noise=true , address=77.90.31.19:4853 , piece_id=MS4HNXUQIITLBMEHZDPJMEH7FSLMBBGKJSIP5BY2BDUF3IE2NKIQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.387549123 (601h15m58.258632158s) [noise=true , address=77.90.31.18:4829 , piece_id=MVRLWV5URMPV57R52J426SXY5ZWQGHOTL42GO2GPEZ4LA7B2JGCQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.394280961 (601h15m58.265364006s) [noise=true , address=5.231.73.4:4033 , piece_id=CCH6AH47YBQPGPF4MPZU2NPT7HPF4D3KT3ZVDITRYF7UW5JUKWDQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.510328564 (601h15m58.381411609s) [noise=true , address=45.91.116.8:4071 , piece_id=3PUQCDJAXSQGEM47BTD2B5HWWNK3PUS6AEVQ63BEC2Q2CKCB4PXQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.545484694 (601h15m58.416567739s) [noise=true , address=45.91.116.11:4131 , piece_id=YAFIDLBM762GWTNKDW62FMNLE4YHCNYW2WU3R4AMO5NELMVOHHHQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.710409562 (601h15m58.581492607s) [noise=true , address=77.90.31.21:4899 , piece_id=DCYPDKTC4S5LG3XEBQWZT3HGFJ63T3UC3PFUDLF7OOJHNMYW4KRQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.981893937 (601h15m58.852976982s) [noise=true , address=145.79.227.44:4212 , piece_id=ZV3XMTVGJR2D5X4EEGGDTGGGXSQI4QNCJZCI3GPR6RTD7JEQWLGA, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.995663946 (601h15m58.866746991s) [noise=true , address=145.79.227.49:4341 , piece_id=6GKQQIBOECLDEEDZNO5AYKX7XMXZYUA6WNGKD3FQSF2Q4O6A4MQA, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:59.009469091 (601h15m58.880552136s) [noise=true , address=145.79.227.40:4064 , piece_id=SEYSDBZHX6SHMFZBGY3XDIYEOTWFCNGG2KNNTCIXYQRZP7FZOVJQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:59.009489950 (601h15m58.880572986s) [seg_pos={0 0}, upload=526987]: Segment reached optimal threshold of 14 pieces.
Dec 30 16:44:59.009662329 (601h15m58.880745374s) [noise=true , address=145.79.227.34:4923 , piece_id=OB3KQEHYXE4BL73EDDQDN2LSVYJYYQQFUSFJ655BFJISLBGOH4XQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VXAxRGiJQor24W7DYSeUQUu1QF7EcX6BQYgfUheACRs2quFFu): protocol: expected piece hash; context canceled; EOF
Dec 30 16:44:59.009668390 (601h15m58.880751425s) [noise=true , address=145.79.227.34:4923 , piece_id=OB3KQEHYXE4BL73EDDQDN2LSVYJYYQQFUSFJ655BFJISLBGOH4XQ, seg_pos={0 0}, upload=526987]: Long tail cancellation due to optimal threshold
Dec 30 16:44:59.009857671 (601h15m58.880940706s) [noise=true , address=145.79.227.33:4903 , piece_id=THSFTSO2Y4QVWIGNUOU4AKFGTSQWQ6FLNIIDISIVNRF7XZZWWPCQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:59.009921232 (601h15m58.881004267s) [noise=true , address=145.79.227.30:4790 , piece_id=5VRI4FYK7J54TCOY7DX6TCQ3634N3TATGDZUZDWRBBLMXCGRVR3Q, seg_pos={0 0}, upload=526987]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kuSMof8EbvsaMCdVxKWjgTT5TfLJnTAeHUf8DKNC3UGQn5q7T): protocol: expected piece hash; context canceled; EOF
Dec 30 16:44:59.009927314 (601h15m58.881010359s) [noise=true , address=145.79.227.30:4790 , piece_id=5VRI4FYK7J54TCOY7DX6TCQ3634N3TATGDZUZDWRBBLMXCGRVR3Q, seg_pos={0 0}, upload=526987]: Long tail cancellation due to optimal threshold
Dec 30 16:44:59.010092589 (601h15m58.881175624s) [noise=true , address=77.90.31.11:4567 , piece_id=NMCHY46TZ6W6EPVLVVCGQXHFERHBLSFB3J55AD32KQSKOVXABO7A, seg_pos={0 0}, upload=526987]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HJVmEuxjrwWCcdwxWYFC5fTULds1kiRH82DiRWipWhRT1Qwdk): protocol: expected piece hash; context canceled; EOF
Dec 30 16:44:59.010101946 (601h15m58.881184981s) [noise=true , address=77.90.31.11:4567 , piece_id=NMCHY46TZ6W6EPVLVVCGQXHFERHBLSFB3J55AD32KQSKOVXABO7A, seg_pos={0 0}, upload=526987]: Long tail cancellation due to optimal threshold
Dec 30 16:44:59.010125410 (601h15m58.881208445s) [seg_pos={0 0}, upload=526987]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil>
Dec 30 16:44:59.010204091 (601h15m58.881287136s) [upload=526987]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 16:44:59.010211274 (601h15m58.881294339s) [upload=526987]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 16:44:59.309660956 (601h15m59.180744001s) [upload=526987]: Done uploading segments...
Dec 30 16:44:59.309679662 (601h15m59.180762697s) [upload=526987]: Upload finished. err: <nil>
Duration: 29.668860111s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [......................................................X...X.] duration:
6.05368425s trace:
https://jaeger.dev.storj.io/trace/13aa859db3bab1bc
FAIL (2025-12-30 17:10:22 UTC)
orbiter warning: context deadline exceeded
Duration: 6.053709949s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
24 out of 60 is failed (threshold=3) [...X..X....X.XXX..XX.X.X..XXX.XX..X.X.......X.X..XX.X.X.X...] duration:
6.023989178s trace:
https://jaeger.dev.storj.io/trace/60c64a4ed511feff
FAIL (2025-12-30 17:10:17 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: 7J6MW2KP7PDK4PR4UWKN27TZVRYTTVB2OUTWOP3QSHNLDCEEAAPQ) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: UR5EMXYBJQZE3QWQOMA4Q4EGTW5YSQZFWXZCMCHY7S6XZSE3DJDQ) context deadline exceeded
======LOGS======
Dec 30 17:07:43.664373604 (601h38m43.535456639s) [upload=527320]: Starting upload
Dec 30 17:07:43.664503591 (601h38m43.535586626s) [upload=527320]: Done starting upload
Dec 30 17:07:43.664596548 (601h38m43.535679603s) [upload=527320]: Uploading segments...
Dec 30 17:07:43.664830463 (601h38m43.535913529s) [upload=527320]: Got next segment. Inline: false
Dec 30 17:07:43.664868426 (601h38m43.535951461s) [upload=527320]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 17:07:43.664873635 (601h38m43.53595668s) [upload=527320]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 17:07:43.912705327 (601h38m43.783788362s) [seg_pos={0 0}, upload=527320]: Begin upload segment...
Dec 30 17:07:43.912759340 (601h38m43.783842385s) [seg_pos={0 0}, upload=527320]: Done begin upload segment.
Dec 30 17:07:43.912809596 (601h38m43.783892641s) [noise=true , address=145.79.227.41:4087 , piece_id=XKNSFDJTAMNYNOSHQRODR43BYHQUCUUXKN5BNX4IN2AZOIQXJPUQ, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.912859631 (601h38m43.783942676s) [noise=true , address=145.79.227.39:4014 , piece_id=OBSTHY6XRVBDBNNY2WQBIKKHZDM5WYSQQXTUTH3HPJHS2M3NQ33Q, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.912913824 (601h38m43.783996859s) [noise=true , address=45.91.116.13:4156 , piece_id=Q6URETP3JATX2X2QNZFZPTZJXNSTSDIRMFMI7IWB667SMWEDOSIA, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.913401693 (601h38m43.784484758s) [noise=true , address=77.90.31.21:4878 , piece_id=FJDJLV23J3EBVPXNCFZICOD5NCDDGAUFE4DQAC6F5VR3G4RJOPOA, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.913826833 (601h38m43.784909868s) [noise=true , address=145.79.227.50:4375 , piece_id=P3VD32XBMCMMIWRXSKXM4MP7MO2EBVNX45CFQ6VE4VCWKVW4ARHA, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.914195945 (601h38m43.78527899s) [noise=true , address=145.79.227.10:4401 , piece_id=GYQLGCGTR2JUCVBJENCDO5X742XLM5IQ5IOUBL5MI6WKATDPRTEQ, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.914624852 (601h38m43.785707897s) [noise=true , address=145.79.227.18:4007 , piece_id=V4AKDVM6FSKQH5W7EU3Z4VAFLBRUWVTLNF52AQWPTWAGXINKHINA, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.914934862 (601h38m43.786017907s) [noise=true , address=77.90.31.13:4626 , piece_id=WG422DPUGAEZ5RWRJQFE6OHR2WN6XB4JMP3RVNQASMTNV2NRKJHQ, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.915253880 (601h38m43.786336915s) [noise=true , address=45.91.116.7:4009 , piece_id=7J6MW2KP7PDK4PR4UWKN27TZVRYTTVB2OUTWOP3QSHNLDCEEAAPQ, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.915558990 (601h38m43.786642025s) [noise=true , address=45.91.116.12:4228 , piece_id=HQPHFGFPMTMGZPA4RDN7WTQVOCYNWKQUJKA4HHP4W2H22GWXPIWQ, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.916736473 (601h38m43.787819518s) [noise=true , address=145.79.227.17:4970 , piece_id=Q5POSLSKIZQ62PP7YC57CMXT6LPEW4TQYAEDDUJLMBKTFMRT2SYQ, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.916879595 (601h38m43.78796264s) [noise=true , address=145.79.227.2:4114 , piece_id=YDKKREVTBHOPAGTQOEIFYE7WPSM34DBEXSPVE2GZELRU7U2V7J7A, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.917867215 (601h38m43.78895025s) [noise=true , address=77.90.31.19:4844 , piece_id=UR5EMXYBJQZE3QWQOMA4Q4EGTW5YSQZFWXZCMCHY7S6XZSE3DJDQ, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.918523956 (601h38m43.789607001s) [noise=true , address=145.79.227.8:4330 , piece_id=JRX2H43ZKEUIIAFBTDKOHQIZ3S3SN3RPCWNN2WHCFS63QAHPUX6A, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.918888360 (601h38m43.789971395s) [noise=true , address=145.79.227.51:4409 , piece_id=EQP42GLRU524RZ3YWXWRXGXUESWWCLBUBFIAQECCTABEWELWBKBQ, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.920412291 (601h38m43.791495326s) [noise=true , address=145.79.227.13:4804 , piece_id=G3QGE5KSKIGSC7OD3D6RMJVIQR2XTRQQSB6GDE755LYQURNLW5QA, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:45.522080803 (601h38m45.393163838s) [upload=527320]: Next returned nil segment
Dec 30 17:07:45.522101643 (601h38m45.393184678s) [upload=527320]: Waiting for error group managing segments...
Dec 30 17:07:45.901475256 (601h38m45.772558291s) [noise=true , address=145.79.227.10:4401 , piece_id=GYQLGCGTR2JUCVBJENCDO5X742XLM5IQ5IOUBL5MI6WKATDPRTEQ, seg_pos={0 0}, upload=527320]: Done uploading piece. err: <nil>
Dec 30 17:07:45.902373236 (601h38m45.773456271s) [noise=true , address=145.79.227.2:4114 , piece_id=YDKKREVTBHOPAGTQOEIFYE7WPSM34DBEXSPVE2GZELRU7U2V7J7A, seg_pos={0 0}, upload=527320]: Done uploading piece. err: <nil>
Dec 30 17:07:45.906142543 (601h38m45.777225578s) [noise=true , address=145.79.227.50:4375 , piece_id=P3VD32XBMCMMIWRXSKXM4MP7MO2EBVNX45CFQ6VE4VCWKVW4ARHA, seg_pos={0 0}, upload=527320]: Done uploading piece. err: <nil>
Dec 30 17:07:45.908434327 (601h38m45.779517362s) [noise=true , address=145.79.227.39:4014 , piece_id=OBSTHY6XRVBDBNNY2WQBIKKHZDM5WYSQQXTUTH3HPJHS2M3NQ33Q, seg_pos={0 0}, upload=527320]: Done uploading piece. err: <nil>
Dec 30 17:07:45.908959187 (601h38m45.780042222s) [noise=true , address=145.79.227.51:4409 , piece_id=EQP42GLRU524RZ3YWXWRXGXUESWWCLBUBFIAQECCTABEWELWBKBQ, seg_pos={0 0}, upload=527320]: Done uploading piece. err: <nil>
Dec 30 17:07:45.929017149 (601h38m45.800100194s) [noise=true , address=145.79.227.8:4330 , piece_id=JRX2H43ZKEUIIAFBTDKOHQIZ3S3SN3RPCWNN2WHCFS63QAHPUX6A, seg_pos={0 0}, upload=527320]: Done uploading piece. err: <nil>
Dec 30 17:07:45.936172594 (601h38m45.807255629s) [noise=true , address=145.79.227.18:4007 , piece_id=V4AKDVM6FSKQH5W7EU3Z4VAFLBRUWVTLNF52AQWPTWAGXINKHINA, seg_pos={0 0}, upload=527320]: Done uploading piece. err: <nil>
Dec 30 17:07:45.937748755 (601h38m45.80883179s) [noise=true , address=145.79.227.41:4087 , piece_id=XKNSFDJTAMNYNOSHQRODR43BYHQUCUUXKN5BNX4IN2AZOIQXJPUQ, seg_pos={0 0}, upload=527320]: Done uploading piece. err: <nil>
Dec 30 17:07:45.940859458 (601h38m45.811942503s) [noise=true , address=145.79.227.13:4804 , piece_id=G3QGE5KSKIGSC7OD3D6RMJVIQR2XTRQQSB6GDE755LYQURNLW5QA, seg_pos={0 0}, upload=527320]: Done uploading piece. err: <nil>
Dec 30 17:07:45.985246526 (601h38m45.856329571s) [noise=true , address=145.79.227.17:4970 , piece_id=Q5POSLSKIZQ62PP7YC57CMXT6LPEW4TQYAEDDUJLMBKTFMRT2SYQ, seg_pos={0 0}, upload=527320]: Done uploading piece. err: <nil>
Dec 30 17:07:46.166004693 (601h38m46.037087748s) [noise=true , address=77.90.31.19:4844 , piece_id=UR5EMXYBJQZE3QWQOMA4Q4EGTW5YSQZFWXZCMCHY7S6XZSE3DJDQ, seg_pos={0 0}, upload=527320]: Done uploading piece. err: <nil>
Dec 30 17:07:46.218612204 (601h38m46.089695249s) [noise=true , address=77.90.31.13:4626 , piece_id=WG422DPUGAEZ5RWRJQFE6OHR2WN6XB4JMP3RVNQASMTNV2NRKJHQ, seg_pos={0 0}, upload=527320]: Done uploading piece. err: <nil>
Dec 30 17:07:46.265695196 (601h38m46.136778231s) [noise=true , address=45.91.116.7:4009 , piece_id=7J6MW2KP7PDK4PR4UWKN27TZVRYTTVB2OUTWOP3QSHNLDCEEAAPQ, seg_pos={0 0}, upload=527320]: Done uploading piece. err: <nil>
Dec 30 17:07:46.265712759 (601h38m46.136795804s) [seg_pos={0 0}, upload=527320]: Segment reached optimal threshold of 13 pieces.
Dec 30 17:07:46.265948578 (601h38m46.137031643s) [noise=true , address=45.91.116.12:4228 , piece_id=HQPHFGFPMTMGZPA4RDN7WTQVOCYNWKQUJKA4HHP4W2H22GWXPIWQ, seg_pos={0 0}, upload=527320]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF
Dec 30 17:07:46.265956693 (601h38m46.137039738s) [noise=true , address=45.91.116.12:4228 , piece_id=HQPHFGFPMTMGZPA4RDN7WTQVOCYNWKQUJKA4HHP4W2H22GWXPIWQ, seg_pos={0 0}, upload=527320]: Long tail cancellation due to optimal threshold
Dec 30 17:07:46.265984967 (601h38m46.137068002s) [noise=true , address=45.91.116.13:4156 , piece_id=Q6URETP3JATX2X2QNZFZPTZJXNSTSDIRMFMI7IWB667SMWEDOSIA, seg_pos={0 0}, upload=527320]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF
Dec 30 17:07:46.265992471 (601h38m46.137075506s) [noise=true , address=45.91.116.13:4156 , piece_id=Q6URETP3JATX2X2QNZFZPTZJXNSTSDIRMFMI7IWB667SMWEDOSIA, seg_pos={0 0}, upload=527320]: Long tail cancellation due to optimal threshold
Dec 30 17:07:46.266129482 (601h38m46.137212517s) [noise=true , address=77.90.31.21:4878 , piece_id=FJDJLV23J3EBVPXNCFZICOD5NCDDGAUFE4DQAC6F5VR3G4RJOPOA, seg_pos={0 0}, upload=527320]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF
Dec 30 17:07:46.266136857 (601h38m46.137219892s) [noise=true , address=77.90.31.21:4878 , piece_id=FJDJLV23J3EBVPXNCFZICOD5NCDDGAUFE4DQAC6F5VR3G4RJOPOA, seg_pos={0 0}, upload=527320]: Long tail cancellation due to optimal threshold
Dec 30 17:07:46.266155642 (601h38m46.137238677s) [seg_pos={0 0}, upload=527320]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 17:07:46.266217931 (601h38m46.137300976s) [upload=527320]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 17:07:46.266222229 (601h38m46.137305264s) [upload=527320]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 17:07:46.559743920 (601h38m46.430826955s) [upload=527320]: Done uploading segments...
Dec 30 17:07:46.559761584 (601h38m46.430844619s) [upload=527320]: Upload finished. err: <nil>
Duration: 6.024019164s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.......................................................X...X] duration:
30.001715018s trace:
https://jaeger.dev.storj.io/trace/13aa859db3bab1bc
FAIL (2025-12-30 17:09:53 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001743643s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
17 out of 60 is failed (threshold=6) [X...XXX..X....XXX..X...XX......X.XX................X..X....X] duration:
5.001526048s trace:
https://jaeger.dev.storj.io/trace/7e2b9a00867448e0
FAIL (2025-12-30 17:09:49 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001548241s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [.......X.............X.......X..............................] duration:
44.010964234s trace:
https://jaeger.dev.storj.io/trace/3f4379d3637c8e4b
FAIL (2025-12-30 17:09:43 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1WBuBdGxdPXi6VsNf4FJpn3LpWoQztM7CV6YjAW1mUDVjAUjn2, Piece ID: MS4HNXUQIITLBMEHZDPJMEH7FSLMBBGKJSIP5BY2BDUF3IE2NKIQ) context deadline exceeded; piecestore: (Node ID: 1soiTQsiMNe5BkNHwwAGykmtP9LtLxncqE7Tv728ak8pjFeshg, Piece ID: BHVQNSBU3LP35Z5WQYTRPJZNNWTDZKWYK2CMRLHSOGBJ4IXGZ5SA) context deadline exceeded; piecestore: (Node ID: 1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, Piece ID: MVRLWV5URMPV57R52J426SXY5ZWQGHOTL42GO2GPEZ4LA7B2JGCQ) context deadline exceeded
======LOGS======
5m56.099625936s) [seg_pos={0 0}, upload=526987]: Begin upload segment...
Dec 30 16:44:56.228605921 (601h15m56.099688967s) [seg_pos={0 0}, upload=526987]: Done begin upload segment.
Dec 30 16:44:56.228736641 (601h15m56.099819686s) [noise=true , address=145.79.227.34:4923 , piece_id=OB3KQEHYXE4BL73EDDQDN2LSVYJYYQQFUSFJ655BFJISLBGOH4XQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.228753061 (601h15m56.099836117s) [noise=true , address=64.49.13.26:4947 , piece_id=BHVQNSBU3LP35Z5WQYTRPJZNNWTDZKWYK2CMRLHSOGBJ4IXGZ5SA, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.229701116 (601h15m56.100784151s) [noise=true , address=45.91.116.8:4071 , piece_id=3PUQCDJAXSQGEM47BTD2B5HWWNK3PUS6AEVQ63BEC2Q2CKCB4PXQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.230399837 (601h15m56.101482882s) [noise=true , address=77.90.31.21:4899 , piece_id=DCYPDKTC4S5LG3XEBQWZT3HGFJ63T3UC3PFUDLF7OOJHNMYW4KRQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.230834926 (601h15m56.101917971s) [noise=true , address=77.90.31.19:4853 , piece_id=MS4HNXUQIITLBMEHZDPJMEH7FSLMBBGKJSIP5BY2BDUF3IE2NKIQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.232511007 (601h15m56.103594112s) [noise=true , address=145.79.227.8:4344 , piece_id=6L5DD7CQJ5JZ7IPFX2IGHNEP7M6BZU2DA5Z5GEOKGPAMVZUXQHOQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.233047428 (601h15m56.104130473s) [noise=true , address=145.79.227.31:4808 , piece_id=MW4UFSJ4XFLXK7ZLWZFWTP2N4QQUS24WP3O44DJWT7AK2WCWAT3Q, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.233557500 (601h15m56.104640545s) [noise=true , address=145.79.227.33:4903 , piece_id=THSFTSO2Y4QVWIGNUOU4AKFGTSQWQ6FLNIIDISIVNRF7XZZWWPCQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.234065937 (601h15m56.105148983s) [noise=true , address=64.49.13.137:4493 , piece_id=QZSPIHMAGKSHSAJYJ6BR26EMTCWFAEPOPC4O65YN6SL6222H4HWQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.234513860 (601h15m56.105596896s) [noise=true , address=64.49.13.27:4965 , piece_id=AQTJD7BXKPMIAGEOGJCY5Y64IVU2SHCZEFJL2NLQAHVX5HR33N7Q, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.236125519 (601h15m56.107208554s) [noise=true , address=77.90.31.11:4567 , piece_id=NMCHY46TZ6W6EPVLVVCGQXHFERHBLSFB3J55AD32KQSKOVXABO7A, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.237179756 (601h15m56.108262791s) [noise=true , address=145.79.227.40:4064 , piece_id=SEYSDBZHX6SHMFZBGY3XDIYEOTWFCNGG2KNNTCIXYQRZP7FZOVJQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.238439144 (601h15m56.109522189s) [noise=true , address=145.79.227.30:4790 , piece_id=5VRI4FYK7J54TCOY7DX6TCQ3634N3TATGDZUZDWRBBLMXCGRVR3Q, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.239316585 (601h15m56.11039962s) [noise=true , address=145.79.227.44:4212 , piece_id=ZV3XMTVGJR2D5X4EEGGDTGGGXSQI4QNCJZCI3GPR6RTD7JEQWLGA, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.242138468 (601h15m56.113221503s) [noise=true , address=45.91.116.11:4131 , piece_id=YAFIDLBM762GWTNKDW62FMNLE4YHCNYW2WU3R4AMO5NELMVOHHHQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.243965547 (601h15m56.115048582s) [noise=true , address=77.90.31.18:4829 , piece_id=MVRLWV5URMPV57R52J426SXY5ZWQGHOTL42GO2GPEZ4LA7B2JGCQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.250805150 (601h15m56.121888185s) [noise=true , address=5.231.73.4:4033 , piece_id=CCH6AH47YBQPGPF4MPZU2NPT7HPF4D3KT3ZVDITRYF7UW5JUKWDQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.254973396 (601h15m56.126056431s) [noise=true , address=145.79.227.49:4341 , piece_id=6GKQQIBOECLDEEDZNO5AYKX7XMXZYUA6WNGKD3FQSF2Q4O6A4MQA, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:57.691193696 (601h15m57.562276731s) [upload=526987]: Next returned nil segment
Dec 30 16:44:57.691206460 (601h15m57.562289495s) [upload=526987]: Waiting for error group managing segments...
Dec 30 16:44:58.085111563 (601h15m57.956194598s) [noise=true , address=64.49.13.137:4493 , piece_id=QZSPIHMAGKSHSAJYJ6BR26EMTCWFAEPOPC4O65YN6SL6222H4HWQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.086614434 (601h15m57.957697479s) [noise=true , address=64.49.13.27:4965 , piece_id=AQTJD7BXKPMIAGEOGJCY5Y64IVU2SHCZEFJL2NLQAHVX5HR33N7Q, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.119421281 (601h15m57.990504316s) [noise=true , address=145.79.227.8:4344 , piece_id=6L5DD7CQJ5JZ7IPFX2IGHNEP7M6BZU2DA5Z5GEOKGPAMVZUXQHOQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.125317667 (601h15m57.996400702s) [noise=true , address=145.79.227.31:4808 , piece_id=MW4UFSJ4XFLXK7ZLWZFWTP2N4QQUS24WP3O44DJWT7AK2WCWAT3Q, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.262371457 (601h15m58.133454502s) [noise=true , address=64.49.13.26:4947 , piece_id=BHVQNSBU3LP35Z5WQYTRPJZNNWTDZKWYK2CMRLHSOGBJ4IXGZ5SA, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.284980216 (601h15m58.156063251s) [noise=true , address=77.90.31.19:4853 , piece_id=MS4HNXUQIITLBMEHZDPJMEH7FSLMBBGKJSIP5BY2BDUF3IE2NKIQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.387549123 (601h15m58.258632158s) [noise=true , address=77.90.31.18:4829 , piece_id=MVRLWV5URMPV57R52J426SXY5ZWQGHOTL42GO2GPEZ4LA7B2JGCQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.394280961 (601h15m58.265364006s) [noise=true , address=5.231.73.4:4033 , piece_id=CCH6AH47YBQPGPF4MPZU2NPT7HPF4D3KT3ZVDITRYF7UW5JUKWDQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.510328564 (601h15m58.381411609s) [noise=true , address=45.91.116.8:4071 , piece_id=3PUQCDJAXSQGEM47BTD2B5HWWNK3PUS6AEVQ63BEC2Q2CKCB4PXQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.545484694 (601h15m58.416567739s) [noise=true , address=45.91.116.11:4131 , piece_id=YAFIDLBM762GWTNKDW62FMNLE4YHCNYW2WU3R4AMO5NELMVOHHHQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.710409562 (601h15m58.581492607s) [noise=true , address=77.90.31.21:4899 , piece_id=DCYPDKTC4S5LG3XEBQWZT3HGFJ63T3UC3PFUDLF7OOJHNMYW4KRQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.981893937 (601h15m58.852976982s) [noise=true , address=145.79.227.44:4212 , piece_id=ZV3XMTVGJR2D5X4EEGGDTGGGXSQI4QNCJZCI3GPR6RTD7JEQWLGA, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.995663946 (601h15m58.866746991s) [noise=true , address=145.79.227.49:4341 , piece_id=6GKQQIBOECLDEEDZNO5AYKX7XMXZYUA6WNGKD3FQSF2Q4O6A4MQA, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:59.009469091 (601h15m58.880552136s) [noise=true , address=145.79.227.40:4064 , piece_id=SEYSDBZHX6SHMFZBGY3XDIYEOTWFCNGG2KNNTCIXYQRZP7FZOVJQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:59.009489950 (601h15m58.880572986s) [seg_pos={0 0}, upload=526987]: Segment reached optimal threshold of 14 pieces.
Dec 30 16:44:59.009662329 (601h15m58.880745374s) [noise=true , address=145.79.227.34:4923 , piece_id=OB3KQEHYXE4BL73EDDQDN2LSVYJYYQQFUSFJ655BFJISLBGOH4XQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VXAxRGiJQor24W7DYSeUQUu1QF7EcX6BQYgfUheACRs2quFFu): protocol: expected piece hash; context canceled; EOF
Dec 30 16:44:59.009668390 (601h15m58.880751425s) [noise=true , address=145.79.227.34:4923 , piece_id=OB3KQEHYXE4BL73EDDQDN2LSVYJYYQQFUSFJ655BFJISLBGOH4XQ, seg_pos={0 0}, upload=526987]: Long tail cancellation due to optimal threshold
Dec 30 16:44:59.009857671 (601h15m58.880940706s) [noise=true , address=145.79.227.33:4903 , piece_id=THSFTSO2Y4QVWIGNUOU4AKFGTSQWQ6FLNIIDISIVNRF7XZZWWPCQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:59.009921232 (601h15m58.881004267s) [noise=true , address=145.79.227.30:4790 , piece_id=5VRI4FYK7J54TCOY7DX6TCQ3634N3TATGDZUZDWRBBLMXCGRVR3Q, seg_pos={0 0}, upload=526987]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kuSMof8EbvsaMCdVxKWjgTT5TfLJnTAeHUf8DKNC3UGQn5q7T): protocol: expected piece hash; context canceled; EOF
Dec 30 16:44:59.009927314 (601h15m58.881010359s) [noise=true , address=145.79.227.30:4790 , piece_id=5VRI4FYK7J54TCOY7DX6TCQ3634N3TATGDZUZDWRBBLMXCGRVR3Q, seg_pos={0 0}, upload=526987]: Long tail cancellation due to optimal threshold
Dec 30 16:44:59.010092589 (601h15m58.881175624s) [noise=true , address=77.90.31.11:4567 , piece_id=NMCHY46TZ6W6EPVLVVCGQXHFERHBLSFB3J55AD32KQSKOVXABO7A, seg_pos={0 0}, upload=526987]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HJVmEuxjrwWCcdwxWYFC5fTULds1kiRH82DiRWipWhRT1Qwdk): protocol: expected piece hash; context canceled; EOF
Dec 30 16:44:59.010101946 (601h15m58.881184981s) [noise=true , address=77.90.31.11:4567 , piece_id=NMCHY46TZ6W6EPVLVVCGQXHFERHBLSFB3J55AD32KQSKOVXABO7A, seg_pos={0 0}, upload=526987]: Long tail cancellation due to optimal threshold
Dec 30 16:44:59.010125410 (601h15m58.881208445s) [seg_pos={0 0}, upload=526987]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil>
Dec 30 16:44:59.010204091 (601h15m58.881287136s) [upload=526987]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 16:44:59.010211274 (601h15m58.881294339s) [upload=526987]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 16:44:59.309660956 (601h15m59.180744001s) [upload=526987]: Done uploading segments...
Dec 30 16:44:59.309679662 (601h15m59.180762697s) [upload=526987]: Upload finished. err: <nil>
Duration: 44.010992789s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
24 out of 60 is failed (threshold=3) [....X..X....X.XXX..XX.X.X..XXX.XX..X.X.......X.X..XX.X.X.X..] duration:
10.741498432s trace:
https://jaeger.dev.storj.io/trace/60c64a4ed511feff
FAIL (2025-12-30 17:09:27 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: 7J6MW2KP7PDK4PR4UWKN27TZVRYTTVB2OUTWOP3QSHNLDCEEAAPQ) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: UR5EMXYBJQZE3QWQOMA4Q4EGTW5YSQZFWXZCMCHY7S6XZSE3DJDQ) context deadline exceeded
======LOGS======
Dec 30 17:07:43.664373604 (601h38m43.535456639s) [upload=527320]: Starting upload
Dec 30 17:07:43.664503591 (601h38m43.535586626s) [upload=527320]: Done starting upload
Dec 30 17:07:43.664596548 (601h38m43.535679603s) [upload=527320]: Uploading segments...
Dec 30 17:07:43.664830463 (601h38m43.535913529s) [upload=527320]: Got next segment. Inline: false
Dec 30 17:07:43.664868426 (601h38m43.535951461s) [upload=527320]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 17:07:43.664873635 (601h38m43.53595668s) [upload=527320]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 17:07:43.912705327 (601h38m43.783788362s) [seg_pos={0 0}, upload=527320]: Begin upload segment...
Dec 30 17:07:43.912759340 (601h38m43.783842385s) [seg_pos={0 0}, upload=527320]: Done begin upload segment.
Dec 30 17:07:43.912809596 (601h38m43.783892641s) [noise=true , address=145.79.227.41:4087 , piece_id=XKNSFDJTAMNYNOSHQRODR43BYHQUCUUXKN5BNX4IN2AZOIQXJPUQ, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.912859631 (601h38m43.783942676s) [noise=true , address=145.79.227.39:4014 , piece_id=OBSTHY6XRVBDBNNY2WQBIKKHZDM5WYSQQXTUTH3HPJHS2M3NQ33Q, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.912913824 (601h38m43.783996859s) [noise=true , address=45.91.116.13:4156 , piece_id=Q6URETP3JATX2X2QNZFZPTZJXNSTSDIRMFMI7IWB667SMWEDOSIA, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.913401693 (601h38m43.784484758s) [noise=true , address=77.90.31.21:4878 , piece_id=FJDJLV23J3EBVPXNCFZICOD5NCDDGAUFE4DQAC6F5VR3G4RJOPOA, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.913826833 (601h38m43.784909868s) [noise=true , address=145.79.227.50:4375 , piece_id=P3VD32XBMCMMIWRXSKXM4MP7MO2EBVNX45CFQ6VE4VCWKVW4ARHA, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.914195945 (601h38m43.78527899s) [noise=true , address=145.79.227.10:4401 , piece_id=GYQLGCGTR2JUCVBJENCDO5X742XLM5IQ5IOUBL5MI6WKATDPRTEQ, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.914624852 (601h38m43.785707897s) [noise=true , address=145.79.227.18:4007 , piece_id=V4AKDVM6FSKQH5W7EU3Z4VAFLBRUWVTLNF52AQWPTWAGXINKHINA, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.914934862 (601h38m43.786017907s) [noise=true , address=77.90.31.13:4626 , piece_id=WG422DPUGAEZ5RWRJQFE6OHR2WN6XB4JMP3RVNQASMTNV2NRKJHQ, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.915253880 (601h38m43.786336915s) [noise=true , address=45.91.116.7:4009 , piece_id=7J6MW2KP7PDK4PR4UWKN27TZVRYTTVB2OUTWOP3QSHNLDCEEAAPQ, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.915558990 (601h38m43.786642025s) [noise=true , address=45.91.116.12:4228 , piece_id=HQPHFGFPMTMGZPA4RDN7WTQVOCYNWKQUJKA4HHP4W2H22GWXPIWQ, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.916736473 (601h38m43.787819518s) [noise=true , address=145.79.227.17:4970 , piece_id=Q5POSLSKIZQ62PP7YC57CMXT6LPEW4TQYAEDDUJLMBKTFMRT2SYQ, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.916879595 (601h38m43.78796264s) [noise=true , address=145.79.227.2:4114 , piece_id=YDKKREVTBHOPAGTQOEIFYE7WPSM34DBEXSPVE2GZELRU7U2V7J7A, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.917867215 (601h38m43.78895025s) [noise=true , address=77.90.31.19:4844 , piece_id=UR5EMXYBJQZE3QWQOMA4Q4EGTW5YSQZFWXZCMCHY7S6XZSE3DJDQ, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.918523956 (601h38m43.789607001s) [noise=true , address=145.79.227.8:4330 , piece_id=JRX2H43ZKEUIIAFBTDKOHQIZ3S3SN3RPCWNN2WHCFS63QAHPUX6A, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.918888360 (601h38m43.789971395s) [noise=true , address=145.79.227.51:4409 , piece_id=EQP42GLRU524RZ3YWXWRXGXUESWWCLBUBFIAQECCTABEWELWBKBQ, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.920412291 (601h38m43.791495326s) [noise=true , address=145.79.227.13:4804 , piece_id=G3QGE5KSKIGSC7OD3D6RMJVIQR2XTRQQSB6GDE755LYQURNLW5QA, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:45.522080803 (601h38m45.393163838s) [upload=527320]: Next returned nil segment
Dec 30 17:07:45.522101643 (601h38m45.393184678s) [upload=527320]: Waiting for error group managing segments...
Dec 30 17:07:45.901475256 (601h38m45.772558291s) [noise=true , address=145.79.227.10:4401 , piece_id=GYQLGCGTR2JUCVBJENCDO5X742XLM5IQ5IOUBL5MI6WKATDPRTEQ, seg_pos={0 0}, upload=527320]: Done uploading piece. err: <nil>
Dec 30 17:07:45.902373236 (601h38m45.773456271s) [noise=true , address=145.79.227.2:4114 , piece_id=YDKKREVTBHOPAGTQOEIFYE7WPSM34DBEXSPVE2GZELRU7U2V7J7A, seg_pos={0 0}, upload=527320]: Done uploading piece. err: <nil>
Dec 30 17:07:45.906142543 (601h38m45.777225578s) [noise=true , address=145.79.227.50:4375 , piece_id=P3VD32XBMCMMIWRXSKXM4MP7MO2EBVNX45CFQ6VE4VCWKVW4ARHA, seg_pos={0 0}, upload=527320]: Done uploading piece. err: <nil>
Dec 30 17:07:45.908434327 (601h38m45.779517362s) [noise=true , address=145.79.227.39:4014 , piece_id=OBSTHY6XRVBDBNNY2WQBIKKHZDM5WYSQQXTUTH3HPJHS2M3NQ33Q, seg_pos={0 0}, upload=527320]: Done uploading piece. err: <nil>
Dec 30 17:07:45.908959187 (601h38m45.780042222s) [noise=true , address=145.79.227.51:4409 , piece_id=EQP42GLRU524RZ3YWXWRXGXUESWWCLBUBFIAQECCTABEWELWBKBQ, seg_pos={0 0}, upload=527320]: Done uploading piece. err: <nil>
Dec 30 17:07:45.929017149 (601h38m45.800100194s) [noise=true , address=145.79.227.8:4330 , piece_id=JRX2H43ZKEUIIAFBTDKOHQIZ3S3SN3RPCWNN2WHCFS63QAHPUX6A, seg_pos={0 0}, upload=527320]: Done uploading piece. err: <nil>
Dec 30 17:07:45.936172594 (601h38m45.807255629s) [noise=true , address=145.79.227.18:4007 , piece_id=V4AKDVM6FSKQH5W7EU3Z4VAFLBRUWVTLNF52AQWPTWAGXINKHINA, seg_pos={0 0}, upload=527320]: Done uploading piece. err: <nil>
Dec 30 17:07:45.937748755 (601h38m45.80883179s) [noise=true , address=145.79.227.41:4087 , piece_id=XKNSFDJTAMNYNOSHQRODR43BYHQUCUUXKN5BNX4IN2AZOIQXJPUQ, seg_pos={0 0}, upload=527320]: Done uploading piece. err: <nil>
Dec 30 17:07:45.940859458 (601h38m45.811942503s) [noise=true , address=145.79.227.13:4804 , piece_id=G3QGE5KSKIGSC7OD3D6RMJVIQR2XTRQQSB6GDE755LYQURNLW5QA, seg_pos={0 0}, upload=527320]: Done uploading piece. err: <nil>
Dec 30 17:07:45.985246526 (601h38m45.856329571s) [noise=true , address=145.79.227.17:4970 , piece_id=Q5POSLSKIZQ62PP7YC57CMXT6LPEW4TQYAEDDUJLMBKTFMRT2SYQ, seg_pos={0 0}, upload=527320]: Done uploading piece. err: <nil>
Dec 30 17:07:46.166004693 (601h38m46.037087748s) [noise=true , address=77.90.31.19:4844 , piece_id=UR5EMXYBJQZE3QWQOMA4Q4EGTW5YSQZFWXZCMCHY7S6XZSE3DJDQ, seg_pos={0 0}, upload=527320]: Done uploading piece. err: <nil>
Dec 30 17:07:46.218612204 (601h38m46.089695249s) [noise=true , address=77.90.31.13:4626 , piece_id=WG422DPUGAEZ5RWRJQFE6OHR2WN6XB4JMP3RVNQASMTNV2NRKJHQ, seg_pos={0 0}, upload=527320]: Done uploading piece. err: <nil>
Dec 30 17:07:46.265695196 (601h38m46.136778231s) [noise=true , address=45.91.116.7:4009 , piece_id=7J6MW2KP7PDK4PR4UWKN27TZVRYTTVB2OUTWOP3QSHNLDCEEAAPQ, seg_pos={0 0}, upload=527320]: Done uploading piece. err: <nil>
Dec 30 17:07:46.265712759 (601h38m46.136795804s) [seg_pos={0 0}, upload=527320]: Segment reached optimal threshold of 13 pieces.
Dec 30 17:07:46.265948578 (601h38m46.137031643s) [noise=true , address=45.91.116.12:4228 , piece_id=HQPHFGFPMTMGZPA4RDN7WTQVOCYNWKQUJKA4HHP4W2H22GWXPIWQ, seg_pos={0 0}, upload=527320]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF
Dec 30 17:07:46.265956693 (601h38m46.137039738s) [noise=true , address=45.91.116.12:4228 , piece_id=HQPHFGFPMTMGZPA4RDN7WTQVOCYNWKQUJKA4HHP4W2H22GWXPIWQ, seg_pos={0 0}, upload=527320]: Long tail cancellation due to optimal threshold
Dec 30 17:07:46.265984967 (601h38m46.137068002s) [noise=true , address=45.91.116.13:4156 , piece_id=Q6URETP3JATX2X2QNZFZPTZJXNSTSDIRMFMI7IWB667SMWEDOSIA, seg_pos={0 0}, upload=527320]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF
Dec 30 17:07:46.265992471 (601h38m46.137075506s) [noise=true , address=45.91.116.13:4156 , piece_id=Q6URETP3JATX2X2QNZFZPTZJXNSTSDIRMFMI7IWB667SMWEDOSIA, seg_pos={0 0}, upload=527320]: Long tail cancellation due to optimal threshold
Dec 30 17:07:46.266129482 (601h38m46.137212517s) [noise=true , address=77.90.31.21:4878 , piece_id=FJDJLV23J3EBVPXNCFZICOD5NCDDGAUFE4DQAC6F5VR3G4RJOPOA, seg_pos={0 0}, upload=527320]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF
Dec 30 17:07:46.266136857 (601h38m46.137219892s) [noise=true , address=77.90.31.21:4878 , piece_id=FJDJLV23J3EBVPXNCFZICOD5NCDDGAUFE4DQAC6F5VR3G4RJOPOA, seg_pos={0 0}, upload=527320]: Long tail cancellation due to optimal threshold
Dec 30 17:07:46.266155642 (601h38m46.137238677s) [seg_pos={0 0}, upload=527320]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 17:07:46.266217931 (601h38m46.137300976s) [upload=527320]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 17:07:46.266222229 (601h38m46.137305264s) [upload=527320]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 17:07:46.559743920 (601h38m46.430826955s) [upload=527320]: Done uploading segments...
Dec 30 17:07:46.559761584 (601h38m46.430844619s) [upload=527320]: Upload finished. err: <nil>
Duration: 10.741526295s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
16 out of 60 is failed (threshold=6) [.X...XXX..X....XXX..X...XX......X.XX................X..X....] duration:
3.624509586s trace:
https://jaeger.dev.storj.io/trace/390f26c24eefb880
FAIL (2025-12-30 17:09:00 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.624530095s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
5.881374099s trace:
https://jaeger.dev.storj.io/trace/729389dc0e4f4bf0
FAIL (2025-12-30 17:08:52 UTC)
orbiter warning: context deadline exceeded
Duration: 5.8814057s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [........X.............X.......X.............................] duration:
37.995521589s trace:
https://jaeger.dev.storj.io/trace/3f4379d3637c8e4b
FAIL (2025-12-30 17:08:51 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1WBuBdGxdPXi6VsNf4FJpn3LpWoQztM7CV6YjAW1mUDVjAUjn2, Piece ID: MS4HNXUQIITLBMEHZDPJMEH7FSLMBBGKJSIP5BY2BDUF3IE2NKIQ) context deadline exceeded; piecestore: (Node ID: 1soiTQsiMNe5BkNHwwAGykmtP9LtLxncqE7Tv728ak8pjFeshg, Piece ID: BHVQNSBU3LP35Z5WQYTRPJZNNWTDZKWYK2CMRLHSOGBJ4IXGZ5SA) context deadline exceeded; piecestore: (Node ID: 1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, Piece ID: MVRLWV5URMPV57R52J426SXY5ZWQGHOTL42GO2GPEZ4LA7B2JGCQ) context deadline exceeded
======LOGS======
5m56.099625936s) [seg_pos={0 0}, upload=526987]: Begin upload segment...
Dec 30 16:44:56.228605921 (601h15m56.099688967s) [seg_pos={0 0}, upload=526987]: Done begin upload segment.
Dec 30 16:44:56.228736641 (601h15m56.099819686s) [noise=true , address=145.79.227.34:4923 , piece_id=OB3KQEHYXE4BL73EDDQDN2LSVYJYYQQFUSFJ655BFJISLBGOH4XQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.228753061 (601h15m56.099836117s) [noise=true , address=64.49.13.26:4947 , piece_id=BHVQNSBU3LP35Z5WQYTRPJZNNWTDZKWYK2CMRLHSOGBJ4IXGZ5SA, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.229701116 (601h15m56.100784151s) [noise=true , address=45.91.116.8:4071 , piece_id=3PUQCDJAXSQGEM47BTD2B5HWWNK3PUS6AEVQ63BEC2Q2CKCB4PXQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.230399837 (601h15m56.101482882s) [noise=true , address=77.90.31.21:4899 , piece_id=DCYPDKTC4S5LG3XEBQWZT3HGFJ63T3UC3PFUDLF7OOJHNMYW4KRQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.230834926 (601h15m56.101917971s) [noise=true , address=77.90.31.19:4853 , piece_id=MS4HNXUQIITLBMEHZDPJMEH7FSLMBBGKJSIP5BY2BDUF3IE2NKIQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.232511007 (601h15m56.103594112s) [noise=true , address=145.79.227.8:4344 , piece_id=6L5DD7CQJ5JZ7IPFX2IGHNEP7M6BZU2DA5Z5GEOKGPAMVZUXQHOQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.233047428 (601h15m56.104130473s) [noise=true , address=145.79.227.31:4808 , piece_id=MW4UFSJ4XFLXK7ZLWZFWTP2N4QQUS24WP3O44DJWT7AK2WCWAT3Q, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.233557500 (601h15m56.104640545s) [noise=true , address=145.79.227.33:4903 , piece_id=THSFTSO2Y4QVWIGNUOU4AKFGTSQWQ6FLNIIDISIVNRF7XZZWWPCQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.234065937 (601h15m56.105148983s) [noise=true , address=64.49.13.137:4493 , piece_id=QZSPIHMAGKSHSAJYJ6BR26EMTCWFAEPOPC4O65YN6SL6222H4HWQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.234513860 (601h15m56.105596896s) [noise=true , address=64.49.13.27:4965 , piece_id=AQTJD7BXKPMIAGEOGJCY5Y64IVU2SHCZEFJL2NLQAHVX5HR33N7Q, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.236125519 (601h15m56.107208554s) [noise=true , address=77.90.31.11:4567 , piece_id=NMCHY46TZ6W6EPVLVVCGQXHFERHBLSFB3J55AD32KQSKOVXABO7A, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.237179756 (601h15m56.108262791s) [noise=true , address=145.79.227.40:4064 , piece_id=SEYSDBZHX6SHMFZBGY3XDIYEOTWFCNGG2KNNTCIXYQRZP7FZOVJQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.238439144 (601h15m56.109522189s) [noise=true , address=145.79.227.30:4790 , piece_id=5VRI4FYK7J54TCOY7DX6TCQ3634N3TATGDZUZDWRBBLMXCGRVR3Q, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.239316585 (601h15m56.11039962s) [noise=true , address=145.79.227.44:4212 , piece_id=ZV3XMTVGJR2D5X4EEGGDTGGGXSQI4QNCJZCI3GPR6RTD7JEQWLGA, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.242138468 (601h15m56.113221503s) [noise=true , address=45.91.116.11:4131 , piece_id=YAFIDLBM762GWTNKDW62FMNLE4YHCNYW2WU3R4AMO5NELMVOHHHQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.243965547 (601h15m56.115048582s) [noise=true , address=77.90.31.18:4829 , piece_id=MVRLWV5URMPV57R52J426SXY5ZWQGHOTL42GO2GPEZ4LA7B2JGCQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.250805150 (601h15m56.121888185s) [noise=true , address=5.231.73.4:4033 , piece_id=CCH6AH47YBQPGPF4MPZU2NPT7HPF4D3KT3ZVDITRYF7UW5JUKWDQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.254973396 (601h15m56.126056431s) [noise=true , address=145.79.227.49:4341 , piece_id=6GKQQIBOECLDEEDZNO5AYKX7XMXZYUA6WNGKD3FQSF2Q4O6A4MQA, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:57.691193696 (601h15m57.562276731s) [upload=526987]: Next returned nil segment
Dec 30 16:44:57.691206460 (601h15m57.562289495s) [upload=526987]: Waiting for error group managing segments...
Dec 30 16:44:58.085111563 (601h15m57.956194598s) [noise=true , address=64.49.13.137:4493 , piece_id=QZSPIHMAGKSHSAJYJ6BR26EMTCWFAEPOPC4O65YN6SL6222H4HWQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.086614434 (601h15m57.957697479s) [noise=true , address=64.49.13.27:4965 , piece_id=AQTJD7BXKPMIAGEOGJCY5Y64IVU2SHCZEFJL2NLQAHVX5HR33N7Q, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.119421281 (601h15m57.990504316s) [noise=true , address=145.79.227.8:4344 , piece_id=6L5DD7CQJ5JZ7IPFX2IGHNEP7M6BZU2DA5Z5GEOKGPAMVZUXQHOQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.125317667 (601h15m57.996400702s) [noise=true , address=145.79.227.31:4808 , piece_id=MW4UFSJ4XFLXK7ZLWZFWTP2N4QQUS24WP3O44DJWT7AK2WCWAT3Q, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.262371457 (601h15m58.133454502s) [noise=true , address=64.49.13.26:4947 , piece_id=BHVQNSBU3LP35Z5WQYTRPJZNNWTDZKWYK2CMRLHSOGBJ4IXGZ5SA, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.284980216 (601h15m58.156063251s) [noise=true , address=77.90.31.19:4853 , piece_id=MS4HNXUQIITLBMEHZDPJMEH7FSLMBBGKJSIP5BY2BDUF3IE2NKIQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.387549123 (601h15m58.258632158s) [noise=true , address=77.90.31.18:4829 , piece_id=MVRLWV5URMPV57R52J426SXY5ZWQGHOTL42GO2GPEZ4LA7B2JGCQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.394280961 (601h15m58.265364006s) [noise=true , address=5.231.73.4:4033 , piece_id=CCH6AH47YBQPGPF4MPZU2NPT7HPF4D3KT3ZVDITRYF7UW5JUKWDQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.510328564 (601h15m58.381411609s) [noise=true , address=45.91.116.8:4071 , piece_id=3PUQCDJAXSQGEM47BTD2B5HWWNK3PUS6AEVQ63BEC2Q2CKCB4PXQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.545484694 (601h15m58.416567739s) [noise=true , address=45.91.116.11:4131 , piece_id=YAFIDLBM762GWTNKDW62FMNLE4YHCNYW2WU3R4AMO5NELMVOHHHQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.710409562 (601h15m58.581492607s) [noise=true , address=77.90.31.21:4899 , piece_id=DCYPDKTC4S5LG3XEBQWZT3HGFJ63T3UC3PFUDLF7OOJHNMYW4KRQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.981893937 (601h15m58.852976982s) [noise=true , address=145.79.227.44:4212 , piece_id=ZV3XMTVGJR2D5X4EEGGDTGGGXSQI4QNCJZCI3GPR6RTD7JEQWLGA, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.995663946 (601h15m58.866746991s) [noise=true , address=145.79.227.49:4341 , piece_id=6GKQQIBOECLDEEDZNO5AYKX7XMXZYUA6WNGKD3FQSF2Q4O6A4MQA, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:59.009469091 (601h15m58.880552136s) [noise=true , address=145.79.227.40:4064 , piece_id=SEYSDBZHX6SHMFZBGY3XDIYEOTWFCNGG2KNNTCIXYQRZP7FZOVJQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:59.009489950 (601h15m58.880572986s) [seg_pos={0 0}, upload=526987]: Segment reached optimal threshold of 14 pieces.
Dec 30 16:44:59.009662329 (601h15m58.880745374s) [noise=true , address=145.79.227.34:4923 , piece_id=OB3KQEHYXE4BL73EDDQDN2LSVYJYYQQFUSFJ655BFJISLBGOH4XQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VXAxRGiJQor24W7DYSeUQUu1QF7EcX6BQYgfUheACRs2quFFu): protocol: expected piece hash; context canceled; EOF
Dec 30 16:44:59.009668390 (601h15m58.880751425s) [noise=true , address=145.79.227.34:4923 , piece_id=OB3KQEHYXE4BL73EDDQDN2LSVYJYYQQFUSFJ655BFJISLBGOH4XQ, seg_pos={0 0}, upload=526987]: Long tail cancellation due to optimal threshold
Dec 30 16:44:59.009857671 (601h15m58.880940706s) [noise=true , address=145.79.227.33:4903 , piece_id=THSFTSO2Y4QVWIGNUOU4AKFGTSQWQ6FLNIIDISIVNRF7XZZWWPCQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:59.009921232 (601h15m58.881004267s) [noise=true , address=145.79.227.30:4790 , piece_id=5VRI4FYK7J54TCOY7DX6TCQ3634N3TATGDZUZDWRBBLMXCGRVR3Q, seg_pos={0 0}, upload=526987]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kuSMof8EbvsaMCdVxKWjgTT5TfLJnTAeHUf8DKNC3UGQn5q7T): protocol: expected piece hash; context canceled; EOF
Dec 30 16:44:59.009927314 (601h15m58.881010359s) [noise=true , address=145.79.227.30:4790 , piece_id=5VRI4FYK7J54TCOY7DX6TCQ3634N3TATGDZUZDWRBBLMXCGRVR3Q, seg_pos={0 0}, upload=526987]: Long tail cancellation due to optimal threshold
Dec 30 16:44:59.010092589 (601h15m58.881175624s) [noise=true , address=77.90.31.11:4567 , piece_id=NMCHY46TZ6W6EPVLVVCGQXHFERHBLSFB3J55AD32KQSKOVXABO7A, seg_pos={0 0}, upload=526987]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HJVmEuxjrwWCcdwxWYFC5fTULds1kiRH82DiRWipWhRT1Qwdk): protocol: expected piece hash; context canceled; EOF
Dec 30 16:44:59.010101946 (601h15m58.881184981s) [noise=true , address=77.90.31.11:4567 , piece_id=NMCHY46TZ6W6EPVLVVCGQXHFERHBLSFB3J55AD32KQSKOVXABO7A, seg_pos={0 0}, upload=526987]: Long tail cancellation due to optimal threshold
Dec 30 16:44:59.010125410 (601h15m58.881208445s) [seg_pos={0 0}, upload=526987]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil>
Dec 30 16:44:59.010204091 (601h15m58.881287136s) [upload=526987]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 16:44:59.010211274 (601h15m58.881294339s) [upload=526987]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 16:44:59.309660956 (601h15m59.180744001s) [upload=526987]: Done uploading segments...
Dec 30 16:44:59.309679662 (601h15m59.180762697s) [upload=526987]: Upload finished. err: <nil>
Duration: 37.995547007s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
24 out of 60 is failed (threshold=3) [.....X..X....X.XXX..XX.X.X..XXX.XX..X.X.......X.X..XX.X.X.X.] duration:
5.910756686s trace:
https://jaeger.dev.storj.io/trace/60c64a4ed511feff
FAIL (2025-12-30 17:08:45 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: 7J6MW2KP7PDK4PR4UWKN27TZVRYTTVB2OUTWOP3QSHNLDCEEAAPQ) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: UR5EMXYBJQZE3QWQOMA4Q4EGTW5YSQZFWXZCMCHY7S6XZSE3DJDQ) context deadline exceeded
======LOGS======
Dec 30 17:07:43.664373604 (601h38m43.535456639s) [upload=527320]: Starting upload
Dec 30 17:07:43.664503591 (601h38m43.535586626s) [upload=527320]: Done starting upload
Dec 30 17:07:43.664596548 (601h38m43.535679603s) [upload=527320]: Uploading segments...
Dec 30 17:07:43.664830463 (601h38m43.535913529s) [upload=527320]: Got next segment. Inline: false
Dec 30 17:07:43.664868426 (601h38m43.535951461s) [upload=527320]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 17:07:43.664873635 (601h38m43.53595668s) [upload=527320]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 17:07:43.912705327 (601h38m43.783788362s) [seg_pos={0 0}, upload=527320]: Begin upload segment...
Dec 30 17:07:43.912759340 (601h38m43.783842385s) [seg_pos={0 0}, upload=527320]: Done begin upload segment.
Dec 30 17:07:43.912809596 (601h38m43.783892641s) [noise=true , address=145.79.227.41:4087 , piece_id=XKNSFDJTAMNYNOSHQRODR43BYHQUCUUXKN5BNX4IN2AZOIQXJPUQ, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.912859631 (601h38m43.783942676s) [noise=true , address=145.79.227.39:4014 , piece_id=OBSTHY6XRVBDBNNY2WQBIKKHZDM5WYSQQXTUTH3HPJHS2M3NQ33Q, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.912913824 (601h38m43.783996859s) [noise=true , address=45.91.116.13:4156 , piece_id=Q6URETP3JATX2X2QNZFZPTZJXNSTSDIRMFMI7IWB667SMWEDOSIA, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.913401693 (601h38m43.784484758s) [noise=true , address=77.90.31.21:4878 , piece_id=FJDJLV23J3EBVPXNCFZICOD5NCDDGAUFE4DQAC6F5VR3G4RJOPOA, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.913826833 (601h38m43.784909868s) [noise=true , address=145.79.227.50:4375 , piece_id=P3VD32XBMCMMIWRXSKXM4MP7MO2EBVNX45CFQ6VE4VCWKVW4ARHA, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.914195945 (601h38m43.78527899s) [noise=true , address=145.79.227.10:4401 , piece_id=GYQLGCGTR2JUCVBJENCDO5X742XLM5IQ5IOUBL5MI6WKATDPRTEQ, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.914624852 (601h38m43.785707897s) [noise=true , address=145.79.227.18:4007 , piece_id=V4AKDVM6FSKQH5W7EU3Z4VAFLBRUWVTLNF52AQWPTWAGXINKHINA, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.914934862 (601h38m43.786017907s) [noise=true , address=77.90.31.13:4626 , piece_id=WG422DPUGAEZ5RWRJQFE6OHR2WN6XB4JMP3RVNQASMTNV2NRKJHQ, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.915253880 (601h38m43.786336915s) [noise=true , address=45.91.116.7:4009 , piece_id=7J6MW2KP7PDK4PR4UWKN27TZVRYTTVB2OUTWOP3QSHNLDCEEAAPQ, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.915558990 (601h38m43.786642025s) [noise=true , address=45.91.116.12:4228 , piece_id=HQPHFGFPMTMGZPA4RDN7WTQVOCYNWKQUJKA4HHP4W2H22GWXPIWQ, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.916736473 (601h38m43.787819518s) [noise=true , address=145.79.227.17:4970 , piece_id=Q5POSLSKIZQ62PP7YC57CMXT6LPEW4TQYAEDDUJLMBKTFMRT2SYQ, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.916879595 (601h38m43.78796264s) [noise=true , address=145.79.227.2:4114 , piece_id=YDKKREVTBHOPAGTQOEIFYE7WPSM34DBEXSPVE2GZELRU7U2V7J7A, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.917867215 (601h38m43.78895025s) [noise=true , address=77.90.31.19:4844 , piece_id=UR5EMXYBJQZE3QWQOMA4Q4EGTW5YSQZFWXZCMCHY7S6XZSE3DJDQ, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.918523956 (601h38m43.789607001s) [noise=true , address=145.79.227.8:4330 , piece_id=JRX2H43ZKEUIIAFBTDKOHQIZ3S3SN3RPCWNN2WHCFS63QAHPUX6A, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.918888360 (601h38m43.789971395s) [noise=true , address=145.79.227.51:4409 , piece_id=EQP42GLRU524RZ3YWXWRXGXUESWWCLBUBFIAQECCTABEWELWBKBQ, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.920412291 (601h38m43.791495326s) [noise=true , address=145.79.227.13:4804 , piece_id=G3QGE5KSKIGSC7OD3D6RMJVIQR2XTRQQSB6GDE755LYQURNLW5QA, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:45.522080803 (601h38m45.393163838s) [upload=527320]: Next returned nil segment
Dec 30 17:07:45.522101643 (601h38m45.393184678s) [upload=527320]: Waiting for error group managing segments...
Dec 30 17:07:45.901475256 (601h38m45.772558291s) [noise=true , address=145.79.227.10:4401 , piece_id=GYQLGCGTR2JUCVBJENCDO5X742XLM5IQ5IOUBL5MI6WKATDPRTEQ, seg_pos={0 0}, upload=527320]: Done uploading piece. err: <nil>
Dec 30 17:07:45.902373236 (601h38m45.773456271s) [noise=true , address=145.79.227.2:4114 , piece_id=YDKKREVTBHOPAGTQOEIFYE7WPSM34DBEXSPVE2GZELRU7U2V7J7A, seg_pos={0 0}, upload=527320]: Done uploading piece. err: <nil>
Dec 30 17:07:45.906142543 (601h38m45.777225578s) [noise=true , address=145.79.227.50:4375 , piece_id=P3VD32XBMCMMIWRXSKXM4MP7MO2EBVNX45CFQ6VE4VCWKVW4ARHA, seg_pos={0 0}, upload=527320]: Done uploading piece. err: <nil>
Dec 30 17:07:45.908434327 (601h38m45.779517362s) [noise=true , address=145.79.227.39:4014 , piece_id=OBSTHY6XRVBDBNNY2WQBIKKHZDM5WYSQQXTUTH3HPJHS2M3NQ33Q, seg_pos={0 0}, upload=527320]: Done uploading piece. err: <nil>
Dec 30 17:07:45.908959187 (601h38m45.780042222s) [noise=true , address=145.79.227.51:4409 , piece_id=EQP42GLRU524RZ3YWXWRXGXUESWWCLBUBFIAQECCTABEWELWBKBQ, seg_pos={0 0}, upload=527320]: Done uploading piece. err: <nil>
Dec 30 17:07:45.929017149 (601h38m45.800100194s) [noise=true , address=145.79.227.8:4330 , piece_id=JRX2H43ZKEUIIAFBTDKOHQIZ3S3SN3RPCWNN2WHCFS63QAHPUX6A, seg_pos={0 0}, upload=527320]: Done uploading piece. err: <nil>
Dec 30 17:07:45.936172594 (601h38m45.807255629s) [noise=true , address=145.79.227.18:4007 , piece_id=V4AKDVM6FSKQH5W7EU3Z4VAFLBRUWVTLNF52AQWPTWAGXINKHINA, seg_pos={0 0}, upload=527320]: Done uploading piece. err: <nil>
Dec 30 17:07:45.937748755 (601h38m45.80883179s) [noise=true , address=145.79.227.41:4087 , piece_id=XKNSFDJTAMNYNOSHQRODR43BYHQUCUUXKN5BNX4IN2AZOIQXJPUQ, seg_pos={0 0}, upload=527320]: Done uploading piece. err: <nil>
Dec 30 17:07:45.940859458 (601h38m45.811942503s) [noise=true , address=145.79.227.13:4804 , piece_id=G3QGE5KSKIGSC7OD3D6RMJVIQR2XTRQQSB6GDE755LYQURNLW5QA, seg_pos={0 0}, upload=527320]: Done uploading piece. err: <nil>
Dec 30 17:07:45.985246526 (601h38m45.856329571s) [noise=true , address=145.79.227.17:4970 , piece_id=Q5POSLSKIZQ62PP7YC57CMXT6LPEW4TQYAEDDUJLMBKTFMRT2SYQ, seg_pos={0 0}, upload=527320]: Done uploading piece. err: <nil>
Dec 30 17:07:46.166004693 (601h38m46.037087748s) [noise=true , address=77.90.31.19:4844 , piece_id=UR5EMXYBJQZE3QWQOMA4Q4EGTW5YSQZFWXZCMCHY7S6XZSE3DJDQ, seg_pos={0 0}, upload=527320]: Done uploading piece. err: <nil>
Dec 30 17:07:46.218612204 (601h38m46.089695249s) [noise=true , address=77.90.31.13:4626 , piece_id=WG422DPUGAEZ5RWRJQFE6OHR2WN6XB4JMP3RVNQASMTNV2NRKJHQ, seg_pos={0 0}, upload=527320]: Done uploading piece. err: <nil>
Dec 30 17:07:46.265695196 (601h38m46.136778231s) [noise=true , address=45.91.116.7:4009 , piece_id=7J6MW2KP7PDK4PR4UWKN27TZVRYTTVB2OUTWOP3QSHNLDCEEAAPQ, seg_pos={0 0}, upload=527320]: Done uploading piece. err: <nil>
Dec 30 17:07:46.265712759 (601h38m46.136795804s) [seg_pos={0 0}, upload=527320]: Segment reached optimal threshold of 13 pieces.
Dec 30 17:07:46.265948578 (601h38m46.137031643s) [noise=true , address=45.91.116.12:4228 , piece_id=HQPHFGFPMTMGZPA4RDN7WTQVOCYNWKQUJKA4HHP4W2H22GWXPIWQ, seg_pos={0 0}, upload=527320]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF
Dec 30 17:07:46.265956693 (601h38m46.137039738s) [noise=true , address=45.91.116.12:4228 , piece_id=HQPHFGFPMTMGZPA4RDN7WTQVOCYNWKQUJKA4HHP4W2H22GWXPIWQ, seg_pos={0 0}, upload=527320]: Long tail cancellation due to optimal threshold
Dec 30 17:07:46.265984967 (601h38m46.137068002s) [noise=true , address=45.91.116.13:4156 , piece_id=Q6URETP3JATX2X2QNZFZPTZJXNSTSDIRMFMI7IWB667SMWEDOSIA, seg_pos={0 0}, upload=527320]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF
Dec 30 17:07:46.265992471 (601h38m46.137075506s) [noise=true , address=45.91.116.13:4156 , piece_id=Q6URETP3JATX2X2QNZFZPTZJXNSTSDIRMFMI7IWB667SMWEDOSIA, seg_pos={0 0}, upload=527320]: Long tail cancellation due to optimal threshold
Dec 30 17:07:46.266129482 (601h38m46.137212517s) [noise=true , address=77.90.31.21:4878 , piece_id=FJDJLV23J3EBVPXNCFZICOD5NCDDGAUFE4DQAC6F5VR3G4RJOPOA, seg_pos={0 0}, upload=527320]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF
Dec 30 17:07:46.266136857 (601h38m46.137219892s) [noise=true , address=77.90.31.21:4878 , piece_id=FJDJLV23J3EBVPXNCFZICOD5NCDDGAUFE4DQAC6F5VR3G4RJOPOA, seg_pos={0 0}, upload=527320]: Long tail cancellation due to optimal threshold
Dec 30 17:07:46.266155642 (601h38m46.137238677s) [seg_pos={0 0}, upload=527320]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 17:07:46.266217931 (601h38m46.137300976s) [upload=527320]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 17:07:46.266222229 (601h38m46.137305264s) [upload=527320]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 17:07:46.559743920 (601h38m46.430826955s) [upload=527320]: Done uploading segments...
Dec 30 17:07:46.559761584 (601h38m46.430844619s) [upload=527320]: Upload finished. err: <nil>
Duration: 5.910776617s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
17 out of 60 is failed (threshold=6) [X.X...XXX..X....XXX..X...XX......X.XX................X..X...] duration:
3.581623263s trace:
https://jaeger.dev.storj.io/trace/390f26c24eefb880
FAIL (2025-12-30 17:08:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.581660685s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
5.99795691s trace:
https://jaeger.dev.storj.io/trace/4fe744e860ffef11
FAIL (2025-12-30 17:08:34 UTC)
orbiter warning: context deadline exceeded
Duration: 5.997983571s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
21.207290205s trace:
https://jaeger.dev.storj.io/trace/729389dc0e4f4bf0
FAIL (2025-12-30 17:08:13 UTC)
orbiter warning: context deadline exceeded
Duration: 21.207317646s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
24 out of 60 is failed (threshold=3) [......X..X....X.XXX..XX.X.X..XXX.XX..X.X.......X.X..XX.X.X.X] duration:
30.002585717s trace:
https://jaeger.dev.storj.io/trace/60c64a4ed511feff
FAIL (2025-12-30 17:08:12 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE, Piece ID: 7J6MW2KP7PDK4PR4UWKN27TZVRYTTVB2OUTWOP3QSHNLDCEEAAPQ) context deadline exceeded; piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: UR5EMXYBJQZE3QWQOMA4Q4EGTW5YSQZFWXZCMCHY7S6XZSE3DJDQ) context deadline exceeded
======LOGS======
Dec 30 17:07:43.664373604 (601h38m43.535456639s) [upload=527320]: Starting upload
Dec 30 17:07:43.664503591 (601h38m43.535586626s) [upload=527320]: Done starting upload
Dec 30 17:07:43.664596548 (601h38m43.535679603s) [upload=527320]: Uploading segments...
Dec 30 17:07:43.664830463 (601h38m43.535913529s) [upload=527320]: Got next segment. Inline: false
Dec 30 17:07:43.664868426 (601h38m43.535951461s) [upload=527320]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 17:07:43.664873635 (601h38m43.53595668s) [upload=527320]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 17:07:43.912705327 (601h38m43.783788362s) [seg_pos={0 0}, upload=527320]: Begin upload segment...
Dec 30 17:07:43.912759340 (601h38m43.783842385s) [seg_pos={0 0}, upload=527320]: Done begin upload segment.
Dec 30 17:07:43.912809596 (601h38m43.783892641s) [noise=true , address=145.79.227.41:4087 , piece_id=XKNSFDJTAMNYNOSHQRODR43BYHQUCUUXKN5BNX4IN2AZOIQXJPUQ, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.912859631 (601h38m43.783942676s) [noise=true , address=145.79.227.39:4014 , piece_id=OBSTHY6XRVBDBNNY2WQBIKKHZDM5WYSQQXTUTH3HPJHS2M3NQ33Q, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.912913824 (601h38m43.783996859s) [noise=true , address=45.91.116.13:4156 , piece_id=Q6URETP3JATX2X2QNZFZPTZJXNSTSDIRMFMI7IWB667SMWEDOSIA, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.913401693 (601h38m43.784484758s) [noise=true , address=77.90.31.21:4878 , piece_id=FJDJLV23J3EBVPXNCFZICOD5NCDDGAUFE4DQAC6F5VR3G4RJOPOA, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.913826833 (601h38m43.784909868s) [noise=true , address=145.79.227.50:4375 , piece_id=P3VD32XBMCMMIWRXSKXM4MP7MO2EBVNX45CFQ6VE4VCWKVW4ARHA, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.914195945 (601h38m43.78527899s) [noise=true , address=145.79.227.10:4401 , piece_id=GYQLGCGTR2JUCVBJENCDO5X742XLM5IQ5IOUBL5MI6WKATDPRTEQ, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.914624852 (601h38m43.785707897s) [noise=true , address=145.79.227.18:4007 , piece_id=V4AKDVM6FSKQH5W7EU3Z4VAFLBRUWVTLNF52AQWPTWAGXINKHINA, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.914934862 (601h38m43.786017907s) [noise=true , address=77.90.31.13:4626 , piece_id=WG422DPUGAEZ5RWRJQFE6OHR2WN6XB4JMP3RVNQASMTNV2NRKJHQ, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.915253880 (601h38m43.786336915s) [noise=true , address=45.91.116.7:4009 , piece_id=7J6MW2KP7PDK4PR4UWKN27TZVRYTTVB2OUTWOP3QSHNLDCEEAAPQ, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.915558990 (601h38m43.786642025s) [noise=true , address=45.91.116.12:4228 , piece_id=HQPHFGFPMTMGZPA4RDN7WTQVOCYNWKQUJKA4HHP4W2H22GWXPIWQ, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.916736473 (601h38m43.787819518s) [noise=true , address=145.79.227.17:4970 , piece_id=Q5POSLSKIZQ62PP7YC57CMXT6LPEW4TQYAEDDUJLMBKTFMRT2SYQ, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.916879595 (601h38m43.78796264s) [noise=true , address=145.79.227.2:4114 , piece_id=YDKKREVTBHOPAGTQOEIFYE7WPSM34DBEXSPVE2GZELRU7U2V7J7A, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.917867215 (601h38m43.78895025s) [noise=true , address=77.90.31.19:4844 , piece_id=UR5EMXYBJQZE3QWQOMA4Q4EGTW5YSQZFWXZCMCHY7S6XZSE3DJDQ, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.918523956 (601h38m43.789607001s) [noise=true , address=145.79.227.8:4330 , piece_id=JRX2H43ZKEUIIAFBTDKOHQIZ3S3SN3RPCWNN2WHCFS63QAHPUX6A, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.918888360 (601h38m43.789971395s) [noise=true , address=145.79.227.51:4409 , piece_id=EQP42GLRU524RZ3YWXWRXGXUESWWCLBUBFIAQECCTABEWELWBKBQ, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:43.920412291 (601h38m43.791495326s) [noise=true , address=145.79.227.13:4804 , piece_id=G3QGE5KSKIGSC7OD3D6RMJVIQR2XTRQQSB6GDE755LYQURNLW5QA, seg_pos={0 0}, upload=527320]: Uploading piece...
Dec 30 17:07:45.522080803 (601h38m45.393163838s) [upload=527320]: Next returned nil segment
Dec 30 17:07:45.522101643 (601h38m45.393184678s) [upload=527320]: Waiting for error group managing segments...
Dec 30 17:07:45.901475256 (601h38m45.772558291s) [noise=true , address=145.79.227.10:4401 , piece_id=GYQLGCGTR2JUCVBJENCDO5X742XLM5IQ5IOUBL5MI6WKATDPRTEQ, seg_pos={0 0}, upload=527320]: Done uploading piece. err: <nil>
Dec 30 17:07:45.902373236 (601h38m45.773456271s) [noise=true , address=145.79.227.2:4114 , piece_id=YDKKREVTBHOPAGTQOEIFYE7WPSM34DBEXSPVE2GZELRU7U2V7J7A, seg_pos={0 0}, upload=527320]: Done uploading piece. err: <nil>
Dec 30 17:07:45.906142543 (601h38m45.777225578s) [noise=true , address=145.79.227.50:4375 , piece_id=P3VD32XBMCMMIWRXSKXM4MP7MO2EBVNX45CFQ6VE4VCWKVW4ARHA, seg_pos={0 0}, upload=527320]: Done uploading piece. err: <nil>
Dec 30 17:07:45.908434327 (601h38m45.779517362s) [noise=true , address=145.79.227.39:4014 , piece_id=OBSTHY6XRVBDBNNY2WQBIKKHZDM5WYSQQXTUTH3HPJHS2M3NQ33Q, seg_pos={0 0}, upload=527320]: Done uploading piece. err: <nil>
Dec 30 17:07:45.908959187 (601h38m45.780042222s) [noise=true , address=145.79.227.51:4409 , piece_id=EQP42GLRU524RZ3YWXWRXGXUESWWCLBUBFIAQECCTABEWELWBKBQ, seg_pos={0 0}, upload=527320]: Done uploading piece. err: <nil>
Dec 30 17:07:45.929017149 (601h38m45.800100194s) [noise=true , address=145.79.227.8:4330 , piece_id=JRX2H43ZKEUIIAFBTDKOHQIZ3S3SN3RPCWNN2WHCFS63QAHPUX6A, seg_pos={0 0}, upload=527320]: Done uploading piece. err: <nil>
Dec 30 17:07:45.936172594 (601h38m45.807255629s) [noise=true , address=145.79.227.18:4007 , piece_id=V4AKDVM6FSKQH5W7EU3Z4VAFLBRUWVTLNF52AQWPTWAGXINKHINA, seg_pos={0 0}, upload=527320]: Done uploading piece. err: <nil>
Dec 30 17:07:45.937748755 (601h38m45.80883179s) [noise=true , address=145.79.227.41:4087 , piece_id=XKNSFDJTAMNYNOSHQRODR43BYHQUCUUXKN5BNX4IN2AZOIQXJPUQ, seg_pos={0 0}, upload=527320]: Done uploading piece. err: <nil>
Dec 30 17:07:45.940859458 (601h38m45.811942503s) [noise=true , address=145.79.227.13:4804 , piece_id=G3QGE5KSKIGSC7OD3D6RMJVIQR2XTRQQSB6GDE755LYQURNLW5QA, seg_pos={0 0}, upload=527320]: Done uploading piece. err: <nil>
Dec 30 17:07:45.985246526 (601h38m45.856329571s) [noise=true , address=145.79.227.17:4970 , piece_id=Q5POSLSKIZQ62PP7YC57CMXT6LPEW4TQYAEDDUJLMBKTFMRT2SYQ, seg_pos={0 0}, upload=527320]: Done uploading piece. err: <nil>
Dec 30 17:07:46.166004693 (601h38m46.037087748s) [noise=true , address=77.90.31.19:4844 , piece_id=UR5EMXYBJQZE3QWQOMA4Q4EGTW5YSQZFWXZCMCHY7S6XZSE3DJDQ, seg_pos={0 0}, upload=527320]: Done uploading piece. err: <nil>
Dec 30 17:07:46.218612204 (601h38m46.089695249s) [noise=true , address=77.90.31.13:4626 , piece_id=WG422DPUGAEZ5RWRJQFE6OHR2WN6XB4JMP3RVNQASMTNV2NRKJHQ, seg_pos={0 0}, upload=527320]: Done uploading piece. err: <nil>
Dec 30 17:07:46.265695196 (601h38m46.136778231s) [noise=true , address=45.91.116.7:4009 , piece_id=7J6MW2KP7PDK4PR4UWKN27TZVRYTTVB2OUTWOP3QSHNLDCEEAAPQ, seg_pos={0 0}, upload=527320]: Done uploading piece. err: <nil>
Dec 30 17:07:46.265712759 (601h38m46.136795804s) [seg_pos={0 0}, upload=527320]: Segment reached optimal threshold of 13 pieces.
Dec 30 17:07:46.265948578 (601h38m46.137031643s) [noise=true , address=45.91.116.12:4228 , piece_id=HQPHFGFPMTMGZPA4RDN7WTQVOCYNWKQUJKA4HHP4W2H22GWXPIWQ, seg_pos={0 0}, upload=527320]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF
Dec 30 17:07:46.265956693 (601h38m46.137039738s) [noise=true , address=45.91.116.12:4228 , piece_id=HQPHFGFPMTMGZPA4RDN7WTQVOCYNWKQUJKA4HHP4W2H22GWXPIWQ, seg_pos={0 0}, upload=527320]: Long tail cancellation due to optimal threshold
Dec 30 17:07:46.265984967 (601h38m46.137068002s) [noise=true , address=45.91.116.13:4156 , piece_id=Q6URETP3JATX2X2QNZFZPTZJXNSTSDIRMFMI7IWB667SMWEDOSIA, seg_pos={0 0}, upload=527320]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: expected piece hash; context canceled; EOF
Dec 30 17:07:46.265992471 (601h38m46.137075506s) [noise=true , address=45.91.116.13:4156 , piece_id=Q6URETP3JATX2X2QNZFZPTZJXNSTSDIRMFMI7IWB667SMWEDOSIA, seg_pos={0 0}, upload=527320]: Long tail cancellation due to optimal threshold
Dec 30 17:07:46.266129482 (601h38m46.137212517s) [noise=true , address=77.90.31.21:4878 , piece_id=FJDJLV23J3EBVPXNCFZICOD5NCDDGAUFE4DQAC6F5VR3G4RJOPOA, seg_pos={0 0}, upload=527320]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF
Dec 30 17:07:46.266136857 (601h38m46.137219892s) [noise=true , address=77.90.31.21:4878 , piece_id=FJDJLV23J3EBVPXNCFZICOD5NCDDGAUFE4DQAC6F5VR3G4RJOPOA, seg_pos={0 0}, upload=527320]: Long tail cancellation due to optimal threshold
Dec 30 17:07:46.266155642 (601h38m46.137238677s) [seg_pos={0 0}, upload=527320]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 17:07:46.266217931 (601h38m46.137300976s) [upload=527320]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 17:07:46.266222229 (601h38m46.137305264s) [upload=527320]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 17:07:46.559743920 (601h38m46.430826955s) [upload=527320]: Done uploading segments...
Dec 30 17:07:46.559761584 (601h38m46.430844619s) [upload=527320]: Upload finished. err: <nil>
Duration: 30.002606556s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
5.890750209s trace:
https://jaeger.dev.storj.io/trace/4fe744e860ffef11
FAIL (2025-12-30 17:07:48 UTC)
orbiter warning: context deadline exceeded
Duration: 5.890798801s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
17 out of 60 is failed (threshold=6) [.X.X...XXX..X....XXX..X...XX......X.XX................X..X..] duration:
3.506034208s trace:
https://jaeger.dev.storj.io/trace/390f26c24eefb880
FAIL (2025-12-30 17:07:44 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.506067442s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [.........X.............X.......X............................] duration:
41.475427708s trace:
https://jaeger.dev.storj.io/trace/3f4379d3637c8e4b
FAIL (2025-12-30 17:07:43 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1WBuBdGxdPXi6VsNf4FJpn3LpWoQztM7CV6YjAW1mUDVjAUjn2, Piece ID: MS4HNXUQIITLBMEHZDPJMEH7FSLMBBGKJSIP5BY2BDUF3IE2NKIQ) context deadline exceeded; piecestore: (Node ID: 1soiTQsiMNe5BkNHwwAGykmtP9LtLxncqE7Tv728ak8pjFeshg, Piece ID: BHVQNSBU3LP35Z5WQYTRPJZNNWTDZKWYK2CMRLHSOGBJ4IXGZ5SA) context deadline exceeded; piecestore: (Node ID: 1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, Piece ID: MVRLWV5URMPV57R52J426SXY5ZWQGHOTL42GO2GPEZ4LA7B2JGCQ) context deadline exceeded
======LOGS======
5m56.099625936s) [seg_pos={0 0}, upload=526987]: Begin upload segment...
Dec 30 16:44:56.228605921 (601h15m56.099688967s) [seg_pos={0 0}, upload=526987]: Done begin upload segment.
Dec 30 16:44:56.228736641 (601h15m56.099819686s) [noise=true , address=145.79.227.34:4923 , piece_id=OB3KQEHYXE4BL73EDDQDN2LSVYJYYQQFUSFJ655BFJISLBGOH4XQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.228753061 (601h15m56.099836117s) [noise=true , address=64.49.13.26:4947 , piece_id=BHVQNSBU3LP35Z5WQYTRPJZNNWTDZKWYK2CMRLHSOGBJ4IXGZ5SA, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.229701116 (601h15m56.100784151s) [noise=true , address=45.91.116.8:4071 , piece_id=3PUQCDJAXSQGEM47BTD2B5HWWNK3PUS6AEVQ63BEC2Q2CKCB4PXQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.230399837 (601h15m56.101482882s) [noise=true , address=77.90.31.21:4899 , piece_id=DCYPDKTC4S5LG3XEBQWZT3HGFJ63T3UC3PFUDLF7OOJHNMYW4KRQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.230834926 (601h15m56.101917971s) [noise=true , address=77.90.31.19:4853 , piece_id=MS4HNXUQIITLBMEHZDPJMEH7FSLMBBGKJSIP5BY2BDUF3IE2NKIQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.232511007 (601h15m56.103594112s) [noise=true , address=145.79.227.8:4344 , piece_id=6L5DD7CQJ5JZ7IPFX2IGHNEP7M6BZU2DA5Z5GEOKGPAMVZUXQHOQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.233047428 (601h15m56.104130473s) [noise=true , address=145.79.227.31:4808 , piece_id=MW4UFSJ4XFLXK7ZLWZFWTP2N4QQUS24WP3O44DJWT7AK2WCWAT3Q, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.233557500 (601h15m56.104640545s) [noise=true , address=145.79.227.33:4903 , piece_id=THSFTSO2Y4QVWIGNUOU4AKFGTSQWQ6FLNIIDISIVNRF7XZZWWPCQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.234065937 (601h15m56.105148983s) [noise=true , address=64.49.13.137:4493 , piece_id=QZSPIHMAGKSHSAJYJ6BR26EMTCWFAEPOPC4O65YN6SL6222H4HWQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.234513860 (601h15m56.105596896s) [noise=true , address=64.49.13.27:4965 , piece_id=AQTJD7BXKPMIAGEOGJCY5Y64IVU2SHCZEFJL2NLQAHVX5HR33N7Q, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.236125519 (601h15m56.107208554s) [noise=true , address=77.90.31.11:4567 , piece_id=NMCHY46TZ6W6EPVLVVCGQXHFERHBLSFB3J55AD32KQSKOVXABO7A, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.237179756 (601h15m56.108262791s) [noise=true , address=145.79.227.40:4064 , piece_id=SEYSDBZHX6SHMFZBGY3XDIYEOTWFCNGG2KNNTCIXYQRZP7FZOVJQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.238439144 (601h15m56.109522189s) [noise=true , address=145.79.227.30:4790 , piece_id=5VRI4FYK7J54TCOY7DX6TCQ3634N3TATGDZUZDWRBBLMXCGRVR3Q, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.239316585 (601h15m56.11039962s) [noise=true , address=145.79.227.44:4212 , piece_id=ZV3XMTVGJR2D5X4EEGGDTGGGXSQI4QNCJZCI3GPR6RTD7JEQWLGA, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.242138468 (601h15m56.113221503s) [noise=true , address=45.91.116.11:4131 , piece_id=YAFIDLBM762GWTNKDW62FMNLE4YHCNYW2WU3R4AMO5NELMVOHHHQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.243965547 (601h15m56.115048582s) [noise=true , address=77.90.31.18:4829 , piece_id=MVRLWV5URMPV57R52J426SXY5ZWQGHOTL42GO2GPEZ4LA7B2JGCQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.250805150 (601h15m56.121888185s) [noise=true , address=5.231.73.4:4033 , piece_id=CCH6AH47YBQPGPF4MPZU2NPT7HPF4D3KT3ZVDITRYF7UW5JUKWDQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.254973396 (601h15m56.126056431s) [noise=true , address=145.79.227.49:4341 , piece_id=6GKQQIBOECLDEEDZNO5AYKX7XMXZYUA6WNGKD3FQSF2Q4O6A4MQA, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:57.691193696 (601h15m57.562276731s) [upload=526987]: Next returned nil segment
Dec 30 16:44:57.691206460 (601h15m57.562289495s) [upload=526987]: Waiting for error group managing segments...
Dec 30 16:44:58.085111563 (601h15m57.956194598s) [noise=true , address=64.49.13.137:4493 , piece_id=QZSPIHMAGKSHSAJYJ6BR26EMTCWFAEPOPC4O65YN6SL6222H4HWQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.086614434 (601h15m57.957697479s) [noise=true , address=64.49.13.27:4965 , piece_id=AQTJD7BXKPMIAGEOGJCY5Y64IVU2SHCZEFJL2NLQAHVX5HR33N7Q, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.119421281 (601h15m57.990504316s) [noise=true , address=145.79.227.8:4344 , piece_id=6L5DD7CQJ5JZ7IPFX2IGHNEP7M6BZU2DA5Z5GEOKGPAMVZUXQHOQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.125317667 (601h15m57.996400702s) [noise=true , address=145.79.227.31:4808 , piece_id=MW4UFSJ4XFLXK7ZLWZFWTP2N4QQUS24WP3O44DJWT7AK2WCWAT3Q, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.262371457 (601h15m58.133454502s) [noise=true , address=64.49.13.26:4947 , piece_id=BHVQNSBU3LP35Z5WQYTRPJZNNWTDZKWYK2CMRLHSOGBJ4IXGZ5SA, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.284980216 (601h15m58.156063251s) [noise=true , address=77.90.31.19:4853 , piece_id=MS4HNXUQIITLBMEHZDPJMEH7FSLMBBGKJSIP5BY2BDUF3IE2NKIQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.387549123 (601h15m58.258632158s) [noise=true , address=77.90.31.18:4829 , piece_id=MVRLWV5URMPV57R52J426SXY5ZWQGHOTL42GO2GPEZ4LA7B2JGCQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.394280961 (601h15m58.265364006s) [noise=true , address=5.231.73.4:4033 , piece_id=CCH6AH47YBQPGPF4MPZU2NPT7HPF4D3KT3ZVDITRYF7UW5JUKWDQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.510328564 (601h15m58.381411609s) [noise=true , address=45.91.116.8:4071 , piece_id=3PUQCDJAXSQGEM47BTD2B5HWWNK3PUS6AEVQ63BEC2Q2CKCB4PXQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.545484694 (601h15m58.416567739s) [noise=true , address=45.91.116.11:4131 , piece_id=YAFIDLBM762GWTNKDW62FMNLE4YHCNYW2WU3R4AMO5NELMVOHHHQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.710409562 (601h15m58.581492607s) [noise=true , address=77.90.31.21:4899 , piece_id=DCYPDKTC4S5LG3XEBQWZT3HGFJ63T3UC3PFUDLF7OOJHNMYW4KRQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.981893937 (601h15m58.852976982s) [noise=true , address=145.79.227.44:4212 , piece_id=ZV3XMTVGJR2D5X4EEGGDTGGGXSQI4QNCJZCI3GPR6RTD7JEQWLGA, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.995663946 (601h15m58.866746991s) [noise=true , address=145.79.227.49:4341 , piece_id=6GKQQIBOECLDEEDZNO5AYKX7XMXZYUA6WNGKD3FQSF2Q4O6A4MQA, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:59.009469091 (601h15m58.880552136s) [noise=true , address=145.79.227.40:4064 , piece_id=SEYSDBZHX6SHMFZBGY3XDIYEOTWFCNGG2KNNTCIXYQRZP7FZOVJQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:59.009489950 (601h15m58.880572986s) [seg_pos={0 0}, upload=526987]: Segment reached optimal threshold of 14 pieces.
Dec 30 16:44:59.009662329 (601h15m58.880745374s) [noise=true , address=145.79.227.34:4923 , piece_id=OB3KQEHYXE4BL73EDDQDN2LSVYJYYQQFUSFJ655BFJISLBGOH4XQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VXAxRGiJQor24W7DYSeUQUu1QF7EcX6BQYgfUheACRs2quFFu): protocol: expected piece hash; context canceled; EOF
Dec 30 16:44:59.009668390 (601h15m58.880751425s) [noise=true , address=145.79.227.34:4923 , piece_id=OB3KQEHYXE4BL73EDDQDN2LSVYJYYQQFUSFJ655BFJISLBGOH4XQ, seg_pos={0 0}, upload=526987]: Long tail cancellation due to optimal threshold
Dec 30 16:44:59.009857671 (601h15m58.880940706s) [noise=true , address=145.79.227.33:4903 , piece_id=THSFTSO2Y4QVWIGNUOU4AKFGTSQWQ6FLNIIDISIVNRF7XZZWWPCQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:59.009921232 (601h15m58.881004267s) [noise=true , address=145.79.227.30:4790 , piece_id=5VRI4FYK7J54TCOY7DX6TCQ3634N3TATGDZUZDWRBBLMXCGRVR3Q, seg_pos={0 0}, upload=526987]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kuSMof8EbvsaMCdVxKWjgTT5TfLJnTAeHUf8DKNC3UGQn5q7T): protocol: expected piece hash; context canceled; EOF
Dec 30 16:44:59.009927314 (601h15m58.881010359s) [noise=true , address=145.79.227.30:4790 , piece_id=5VRI4FYK7J54TCOY7DX6TCQ3634N3TATGDZUZDWRBBLMXCGRVR3Q, seg_pos={0 0}, upload=526987]: Long tail cancellation due to optimal threshold
Dec 30 16:44:59.010092589 (601h15m58.881175624s) [noise=true , address=77.90.31.11:4567 , piece_id=NMCHY46TZ6W6EPVLVVCGQXHFERHBLSFB3J55AD32KQSKOVXABO7A, seg_pos={0 0}, upload=526987]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HJVmEuxjrwWCcdwxWYFC5fTULds1kiRH82DiRWipWhRT1Qwdk): protocol: expected piece hash; context canceled; EOF
Dec 30 16:44:59.010101946 (601h15m58.881184981s) [noise=true , address=77.90.31.11:4567 , piece_id=NMCHY46TZ6W6EPVLVVCGQXHFERHBLSFB3J55AD32KQSKOVXABO7A, seg_pos={0 0}, upload=526987]: Long tail cancellation due to optimal threshold
Dec 30 16:44:59.010125410 (601h15m58.881208445s) [seg_pos={0 0}, upload=526987]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil>
Dec 30 16:44:59.010204091 (601h15m58.881287136s) [upload=526987]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 16:44:59.010211274 (601h15m58.881294339s) [upload=526987]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 16:44:59.309660956 (601h15m59.180744001s) [upload=526987]: Done uploading segments...
Dec 30 16:44:59.309679662 (601h15m59.180762697s) [upload=526987]: Upload finished. err: <nil>
Duration: 41.475453979s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
23 out of 60 is failed (threshold=3) [.......X..X....X.XXX..XX.X.X..XXX.XX..X.X.......X.X..XX.X.X.] duration:
6.282758045s trace:
https://jaeger.dev.storj.io/trace/42f63950ef3402fa
FAIL (2025-12-30 17:07:27 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: TGHI3F376RWOGTT7ZDJABFCHYF66PTTCZ5SNNVUOPPMP4DXNOJKA) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: 67IB5J4HE7CNTANI6ZX4U7ZIKVCV4P5IBC62J6EPDFIMV4B6TXJQ) context deadline exceeded
======LOGS======
Dec 30 17:06:51.846185314 (601h37m51.71726836s) [upload=527305]: Starting upload
Dec 30 17:06:51.846298870 (601h37m51.717381906s) [upload=527305]: Done starting upload
Dec 30 17:06:51.846342865 (601h37m51.71742591s) [upload=527305]: Uploading segments...
Dec 30 17:06:51.846617958 (601h37m51.717701003s) [upload=527305]: Got next segment. Inline: false
Dec 30 17:06:51.846668064 (601h37m51.717751109s) [upload=527305]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 17:06:51.846673544 (601h37m51.717756589s) [upload=527305]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 17:06:52.097969943 (601h37m51.969052988s) [seg_pos={0 0}, upload=527305]: Begin upload segment...
Dec 30 17:06:52.098017504 (601h37m51.969100539s) [seg_pos={0 0}, upload=527305]: Done begin upload segment.
Dec 30 17:06:52.098034917 (601h37m51.969117952s) [noise=true , address=45.91.116.8:4048 , piece_id=3ZBNNFCD7B55RBVGPWP4CAGHT47RXCMLJLCDI64TTZMHC6GZPZ2A, seg_pos={0 0}, upload=527305]: Uploading piece...
Dec 30 17:06:52.098057560 (601h37m51.969140595s) [noise=true , address=45.91.116.9:4262 , piece_id=3RXCEA5AOYR25CVEQP27AR3HTLY7NYXJKFQVK5DLR6ENSQFHZ2LA, seg_pos={0 0}, upload=527305]: Uploading piece...
Dec 30 17:06:52.098440259 (601h37m51.969523294s) [noise=true , address=45.91.116.14:4192 , piece_id=67IB5J4HE7CNTANI6ZX4U7ZIKVCV4P5IBC62J6EPDFIMV4B6TXJQ, seg_pos={0 0}, upload=527305]: Uploading piece...
Dec 30 17:06:52.098702638 (601h37m51.969785673s) [noise=true , address=145.79.227.18:4006 , piece_id=Q63SV43LFUACDGMHDUJ74VJDCOVGESKO3BE3XPD6TKN5YDACEVVA, seg_pos={0 0}, upload=527305]: Uploading piece...
Dec 30 17:06:52.098948697 (601h37m51.970031732s) [noise=true , address=145.79.227.4:4185 , piece_id=5CJUKD32JKIQJRDDVGNQ35KD5ZT6ORN2PR3PIB7UZ667U75XMXMA, seg_pos={0 0}, upload=527305]: Uploading piece...
Dec 30 17:06:52.099055610 (601h37m51.970138645s) [noise=true , address=145.79.227.42:4122 , piece_id=FVXSWXRUY4AUGVBUJHG3W7ZR5R2LAVXDGMK7Z2FTPB4GZ46VKKAQ, seg_pos={0 0}, upload=527305]: Uploading piece...
Dec 30 17:06:52.099307510 (601h37m51.970390546s) [noise=true , address=145.79.227.51:4410 , piece_id=PDKQJRTE2RQUXWWMKU3ZBJEJVEJALFRUL5PV6CGR4RSFAJFCUTHA, seg_pos={0 0}, upload=527305]: Uploading piece...
Dec 30 17:06:52.099526818 (601h37m51.970609853s) [noise=true , address=145.79.227.2:4114 , piece_id=LTVLXVBY2Y46FPRIO2JFLPZ6TTLKVJIES2H5QHIXBSEXWMMAZLVQ, seg_pos={0 0}, upload=527305]: Uploading piece...
Dec 30 17:06:52.099962256 (601h37m51.971045291s) [noise=true , address=145.79.227.15:4875 , piece_id=NXXFDM6PISZNKGYVGIERVUI47K3QKSXRKKSWFE4MQ3HQAT3OPR4Q, seg_pos={0 0}, upload=527305]: Uploading piece...
Dec 30 17:06:52.100321241 (601h37m51.971404276s) [noise=true , address=45.91.116.4:4940 , piece_id=3JP6EWJDZWUU5I7COBD57WCVFYTO7D33CDMZ23PA3TAE4LXGWABQ, seg_pos={0 0}, upload=527305]: Uploading piece...
Dec 30 17:06:52.100436279 (601h37m51.971519325s) [noise=true , address=77.90.31.18:4808 , piece_id=MGEWL2WO55SAQGAWVX3ZUFTSOUX3XD3VRCHQ5GMU4GAXOGGPFCXA, seg_pos={0 0}, upload=527305]: Uploading piece...
Dec 30 17:06:52.100829277 (601h37m51.971912323s) [noise=true , address=145.79.227.44:4194 , piece_id=4ISX6KXDG35D5EGEJKLDYPLUZO5SZPRHVVAC7NRUXYUKAWPC76IQ, seg_pos={0 0}, upload=527305]: Uploading piece...
Dec 30 17:06:52.101238477 (601h37m51.972321512s) [noise=true , address=145.79.227.47:4265 , piece_id=YMAMVQ6TJQ5EBQEPL6DEPIMIDEKIQPZG6YU5YTII2UNYGJU2YJ6A, seg_pos={0 0}, upload=527305]: Uploading piece...
Dec 30 17:06:52.101263535 (601h37m51.97234658s) [noise=true , address=77.90.31.21:4878 , piece_id=TGHI3F376RWOGTT7ZDJABFCHYF66PTTCZ5SNNVUOPPMP4DXNOJKA, seg_pos={0 0}, upload=527305]: Uploading piece...
Dec 30 17:06:52.101617920 (601h37m51.972700955s) [noise=true , address=145.79.227.46:4603 , piece_id=ANOTTFP7UGHWT6SKMW6OJD4GI5A4FMFAVQQFRW53JDBH6LMRN2SQ, seg_pos={0 0}, upload=527305]: Uploading piece...
Dec 30 17:06:52.102005958 (601h37m51.973088993s) [noise=true , address=145.79.227.17:4970 , piece_id=WXOSUKS4GXS2HH5L75XYDGOEQI27ONJ7GI4TIZNUAN2X5XLYFEWA, seg_pos={0 0}, upload=527305]: Uploading piece...
Dec 30 17:06:53.742945475 (601h37m53.61402851s) [upload=527305]: Next returned nil segment
Dec 30 17:06:53.742961135 (601h37m53.61404417s) [upload=527305]: Waiting for error group managing segments...
Dec 30 17:06:54.072901582 (601h37m53.943984627s) [noise=true , address=145.79.227.51:4410 , piece_id=PDKQJRTE2RQUXWWMKU3ZBJEJVEJALFRUL5PV6CGR4RSFAJFCUTHA, seg_pos={0 0}, upload=527305]: Done uploading piece. err: <nil>
Dec 30 17:06:54.082865995 (601h37m53.95394904s) [noise=true , address=145.79.227.46:4603 , piece_id=ANOTTFP7UGHWT6SKMW6OJD4GI5A4FMFAVQQFRW53JDBH6LMRN2SQ, seg_pos={0 0}, upload=527305]: Done uploading piece. err: <nil>
Dec 30 17:06:54.085784171 (601h37m53.956867206s) [noise=true , address=145.79.227.4:4185 , piece_id=5CJUKD32JKIQJRDDVGNQ35KD5ZT6ORN2PR3PIB7UZ667U75XMXMA, seg_pos={0 0}, upload=527305]: Done uploading piece. err: <nil>
Dec 30 17:06:54.086303410 (601h37m53.957386445s) [noise=true , address=145.79.227.17:4970 , piece_id=WXOSUKS4GXS2HH5L75XYDGOEQI27ONJ7GI4TIZNUAN2X5XLYFEWA, seg_pos={0 0}, upload=527305]: Done uploading piece. err: <nil>
Dec 30 17:06:54.086389444 (601h37m53.957472489s) [noise=true , address=145.79.227.47:4265 , piece_id=YMAMVQ6TJQ5EBQEPL6DEPIMIDEKIQPZG6YU5YTII2UNYGJU2YJ6A, seg_pos={0 0}, upload=527305]: Done uploading piece. err: <nil>
Dec 30 17:06:54.097280211 (601h37m53.968363256s) [noise=true , address=145.79.227.2:4114 , piece_id=LTVLXVBY2Y46FPRIO2JFLPZ6TTLKVJIES2H5QHIXBSEXWMMAZLVQ, seg_pos={0 0}, upload=527305]: Done uploading piece. err: <nil>
Dec 30 17:06:54.106890119 (601h37m53.977973154s) [noise=true , address=145.79.227.42:4122 , piece_id=FVXSWXRUY4AUGVBUJHG3W7ZR5R2LAVXDGMK7Z2FTPB4GZ46VKKAQ, seg_pos={0 0}, upload=527305]: Done uploading piece. err: <nil>
Dec 30 17:06:54.112551680 (601h37m53.983634715s) [noise=true , address=145.79.227.15:4875 , piece_id=NXXFDM6PISZNKGYVGIERVUI47K3QKSXRKKSWFE4MQ3HQAT3OPR4Q, seg_pos={0 0}, upload=527305]: Done uploading piece. err: <nil>
Dec 30 17:06:54.113494676 (601h37m53.984577721s) [noise=true , address=145.79.227.18:4006 , piece_id=Q63SV43LFUACDGMHDUJ74VJDCOVGESKO3BE3XPD6TKN5YDACEVVA, seg_pos={0 0}, upload=527305]: Done uploading piece. err: <nil>
Dec 30 17:06:54.390425606 (601h37m54.261508661s) [noise=true , address=77.90.31.21:4878 , piece_id=TGHI3F376RWOGTT7ZDJABFCHYF66PTTCZ5SNNVUOPPMP4DXNOJKA, seg_pos={0 0}, upload=527305]: Done uploading piece. err: <nil>
Dec 30 17:06:54.419226276 (601h37m54.290309321s) [noise=true , address=45.91.116.9:4262 , piece_id=3RXCEA5AOYR25CVEQP27AR3HTLY7NYXJKFQVK5DLR6ENSQFHZ2LA, seg_pos={0 0}, upload=527305]: Done uploading piece. err: <nil>
Dec 30 17:06:54.471359594 (601h37m54.342442629s) [noise=true , address=45.91.116.4:4940 , piece_id=3JP6EWJDZWUU5I7COBD57WCVFYTO7D33CDMZ23PA3TAE4LXGWABQ, seg_pos={0 0}, upload=527305]: Done uploading piece. err: <nil>
Dec 30 17:06:54.526497273 (601h37m54.397580318s) [noise=true , address=45.91.116.14:4192 , piece_id=67IB5J4HE7CNTANI6ZX4U7ZIKVCV4P5IBC62J6EPDFIMV4B6TXJQ, seg_pos={0 0}, upload=527305]: Done uploading piece. err: <nil>
Dec 30 17:06:54.526521899 (601h37m54.397604934s) [seg_pos={0 0}, upload=527305]: Segment reached optimal threshold of 13 pieces.
Dec 30 17:06:54.526837310 (601h37m54.397920366s) [noise=true , address=45.91.116.8:4048 , piece_id=3ZBNNFCD7B55RBVGPWP4CAGHT47RXCMLJLCDI64TTZMHC6GZPZ2A, seg_pos={0 0}, upload=527305]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF
Dec 30 17:06:54.526886303 (601h37m54.397969338s) [noise=true , address=45.91.116.8:4048 , piece_id=3ZBNNFCD7B55RBVGPWP4CAGHT47RXCMLJLCDI64TTZMHC6GZPZ2A, seg_pos={0 0}, upload=527305]: Long tail cancellation due to optimal threshold
Dec 30 17:06:54.526959072 (601h37m54.398042117s) [noise=true , address=145.79.227.44:4194 , piece_id=4ISX6KXDG35D5EGEJKLDYPLUZO5SZPRHVVAC7NRUXYUKAWPC76IQ, seg_pos={0 0}, upload=527305]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF
Dec 30 17:06:54.526974280 (601h37m54.398057315s) [noise=true , address=145.79.227.44:4194 , piece_id=4ISX6KXDG35D5EGEJKLDYPLUZO5SZPRHVVAC7NRUXYUKAWPC76IQ, seg_pos={0 0}, upload=527305]: Long tail cancellation due to optimal threshold
Dec 30 17:06:54.527120540 (601h37m54.398203575s) [noise=true , address=77.90.31.18:4808 , piece_id=MGEWL2WO55SAQGAWVX3ZUFTSOUX3XD3VRCHQ5GMU4GAXOGGPFCXA, seg_pos={0 0}, upload=527305]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF
Dec 30 17:06:54.527125990 (601h37m54.398209035s) [noise=true , address=77.90.31.18:4808 , piece_id=MGEWL2WO55SAQGAWVX3ZUFTSOUX3XD3VRCHQ5GMU4GAXOGGPFCXA, seg_pos={0 0}, upload=527305]: Long tail cancellation due to optimal threshold
Dec 30 17:06:54.527140998 (601h37m54.398224033s) [seg_pos={0 0}, upload=527305]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 17:06:54.527218225 (601h37m54.39830126s) [upload=527305]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 17:06:54.527231421 (601h37m54.398314456s) [upload=527305]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 17:06:54.825252568 (601h37m54.696335603s) [upload=527305]: Done uploading segments...
Dec 30 17:06:54.825275332 (601h37m54.696358367s) [upload=527305]: Upload finished. err: <nil>
Duration: 6.282776859s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
23 out of 60 is failed (threshold=3) [........X..X....X.XXX..XX.X.X..XXX.XX..X.X.......X.X..XX.X.X] duration:
30.003350069s trace:
https://jaeger.dev.storj.io/trace/42f63950ef3402fa
FAIL (2025-12-30 17:07:21 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: TGHI3F376RWOGTT7ZDJABFCHYF66PTTCZ5SNNVUOPPMP4DXNOJKA) context deadline exceeded; piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: 67IB5J4HE7CNTANI6ZX4U7ZIKVCV4P5IBC62J6EPDFIMV4B6TXJQ) context deadline exceeded
======LOGS======
Dec 30 17:06:51.846185314 (601h37m51.71726836s) [upload=527305]: Starting upload
Dec 30 17:06:51.846298870 (601h37m51.717381906s) [upload=527305]: Done starting upload
Dec 30 17:06:51.846342865 (601h37m51.71742591s) [upload=527305]: Uploading segments...
Dec 30 17:06:51.846617958 (601h37m51.717701003s) [upload=527305]: Got next segment. Inline: false
Dec 30 17:06:51.846668064 (601h37m51.717751109s) [upload=527305]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 17:06:51.846673544 (601h37m51.717756589s) [upload=527305]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 17:06:52.097969943 (601h37m51.969052988s) [seg_pos={0 0}, upload=527305]: Begin upload segment...
Dec 30 17:06:52.098017504 (601h37m51.969100539s) [seg_pos={0 0}, upload=527305]: Done begin upload segment.
Dec 30 17:06:52.098034917 (601h37m51.969117952s) [noise=true , address=45.91.116.8:4048 , piece_id=3ZBNNFCD7B55RBVGPWP4CAGHT47RXCMLJLCDI64TTZMHC6GZPZ2A, seg_pos={0 0}, upload=527305]: Uploading piece...
Dec 30 17:06:52.098057560 (601h37m51.969140595s) [noise=true , address=45.91.116.9:4262 , piece_id=3RXCEA5AOYR25CVEQP27AR3HTLY7NYXJKFQVK5DLR6ENSQFHZ2LA, seg_pos={0 0}, upload=527305]: Uploading piece...
Dec 30 17:06:52.098440259 (601h37m51.969523294s) [noise=true , address=45.91.116.14:4192 , piece_id=67IB5J4HE7CNTANI6ZX4U7ZIKVCV4P5IBC62J6EPDFIMV4B6TXJQ, seg_pos={0 0}, upload=527305]: Uploading piece...
Dec 30 17:06:52.098702638 (601h37m51.969785673s) [noise=true , address=145.79.227.18:4006 , piece_id=Q63SV43LFUACDGMHDUJ74VJDCOVGESKO3BE3XPD6TKN5YDACEVVA, seg_pos={0 0}, upload=527305]: Uploading piece...
Dec 30 17:06:52.098948697 (601h37m51.970031732s) [noise=true , address=145.79.227.4:4185 , piece_id=5CJUKD32JKIQJRDDVGNQ35KD5ZT6ORN2PR3PIB7UZ667U75XMXMA, seg_pos={0 0}, upload=527305]: Uploading piece...
Dec 30 17:06:52.099055610 (601h37m51.970138645s) [noise=true , address=145.79.227.42:4122 , piece_id=FVXSWXRUY4AUGVBUJHG3W7ZR5R2LAVXDGMK7Z2FTPB4GZ46VKKAQ, seg_pos={0 0}, upload=527305]: Uploading piece...
Dec 30 17:06:52.099307510 (601h37m51.970390546s) [noise=true , address=145.79.227.51:4410 , piece_id=PDKQJRTE2RQUXWWMKU3ZBJEJVEJALFRUL5PV6CGR4RSFAJFCUTHA, seg_pos={0 0}, upload=527305]: Uploading piece...
Dec 30 17:06:52.099526818 (601h37m51.970609853s) [noise=true , address=145.79.227.2:4114 , piece_id=LTVLXVBY2Y46FPRIO2JFLPZ6TTLKVJIES2H5QHIXBSEXWMMAZLVQ, seg_pos={0 0}, upload=527305]: Uploading piece...
Dec 30 17:06:52.099962256 (601h37m51.971045291s) [noise=true , address=145.79.227.15:4875 , piece_id=NXXFDM6PISZNKGYVGIERVUI47K3QKSXRKKSWFE4MQ3HQAT3OPR4Q, seg_pos={0 0}, upload=527305]: Uploading piece...
Dec 30 17:06:52.100321241 (601h37m51.971404276s) [noise=true , address=45.91.116.4:4940 , piece_id=3JP6EWJDZWUU5I7COBD57WCVFYTO7D33CDMZ23PA3TAE4LXGWABQ, seg_pos={0 0}, upload=527305]: Uploading piece...
Dec 30 17:06:52.100436279 (601h37m51.971519325s) [noise=true , address=77.90.31.18:4808 , piece_id=MGEWL2WO55SAQGAWVX3ZUFTSOUX3XD3VRCHQ5GMU4GAXOGGPFCXA, seg_pos={0 0}, upload=527305]: Uploading piece...
Dec 30 17:06:52.100829277 (601h37m51.971912323s) [noise=true , address=145.79.227.44:4194 , piece_id=4ISX6KXDG35D5EGEJKLDYPLUZO5SZPRHVVAC7NRUXYUKAWPC76IQ, seg_pos={0 0}, upload=527305]: Uploading piece...
Dec 30 17:06:52.101238477 (601h37m51.972321512s) [noise=true , address=145.79.227.47:4265 , piece_id=YMAMVQ6TJQ5EBQEPL6DEPIMIDEKIQPZG6YU5YTII2UNYGJU2YJ6A, seg_pos={0 0}, upload=527305]: Uploading piece...
Dec 30 17:06:52.101263535 (601h37m51.97234658s) [noise=true , address=77.90.31.21:4878 , piece_id=TGHI3F376RWOGTT7ZDJABFCHYF66PTTCZ5SNNVUOPPMP4DXNOJKA, seg_pos={0 0}, upload=527305]: Uploading piece...
Dec 30 17:06:52.101617920 (601h37m51.972700955s) [noise=true , address=145.79.227.46:4603 , piece_id=ANOTTFP7UGHWT6SKMW6OJD4GI5A4FMFAVQQFRW53JDBH6LMRN2SQ, seg_pos={0 0}, upload=527305]: Uploading piece...
Dec 30 17:06:52.102005958 (601h37m51.973088993s) [noise=true , address=145.79.227.17:4970 , piece_id=WXOSUKS4GXS2HH5L75XYDGOEQI27ONJ7GI4TIZNUAN2X5XLYFEWA, seg_pos={0 0}, upload=527305]: Uploading piece...
Dec 30 17:06:53.742945475 (601h37m53.61402851s) [upload=527305]: Next returned nil segment
Dec 30 17:06:53.742961135 (601h37m53.61404417s) [upload=527305]: Waiting for error group managing segments...
Dec 30 17:06:54.072901582 (601h37m53.943984627s) [noise=true , address=145.79.227.51:4410 , piece_id=PDKQJRTE2RQUXWWMKU3ZBJEJVEJALFRUL5PV6CGR4RSFAJFCUTHA, seg_pos={0 0}, upload=527305]: Done uploading piece. err: <nil>
Dec 30 17:06:54.082865995 (601h37m53.95394904s) [noise=true , address=145.79.227.46:4603 , piece_id=ANOTTFP7UGHWT6SKMW6OJD4GI5A4FMFAVQQFRW53JDBH6LMRN2SQ, seg_pos={0 0}, upload=527305]: Done uploading piece. err: <nil>
Dec 30 17:06:54.085784171 (601h37m53.956867206s) [noise=true , address=145.79.227.4:4185 , piece_id=5CJUKD32JKIQJRDDVGNQ35KD5ZT6ORN2PR3PIB7UZ667U75XMXMA, seg_pos={0 0}, upload=527305]: Done uploading piece. err: <nil>
Dec 30 17:06:54.086303410 (601h37m53.957386445s) [noise=true , address=145.79.227.17:4970 , piece_id=WXOSUKS4GXS2HH5L75XYDGOEQI27ONJ7GI4TIZNUAN2X5XLYFEWA, seg_pos={0 0}, upload=527305]: Done uploading piece. err: <nil>
Dec 30 17:06:54.086389444 (601h37m53.957472489s) [noise=true , address=145.79.227.47:4265 , piece_id=YMAMVQ6TJQ5EBQEPL6DEPIMIDEKIQPZG6YU5YTII2UNYGJU2YJ6A, seg_pos={0 0}, upload=527305]: Done uploading piece. err: <nil>
Dec 30 17:06:54.097280211 (601h37m53.968363256s) [noise=true , address=145.79.227.2:4114 , piece_id=LTVLXVBY2Y46FPRIO2JFLPZ6TTLKVJIES2H5QHIXBSEXWMMAZLVQ, seg_pos={0 0}, upload=527305]: Done uploading piece. err: <nil>
Dec 30 17:06:54.106890119 (601h37m53.977973154s) [noise=true , address=145.79.227.42:4122 , piece_id=FVXSWXRUY4AUGVBUJHG3W7ZR5R2LAVXDGMK7Z2FTPB4GZ46VKKAQ, seg_pos={0 0}, upload=527305]: Done uploading piece. err: <nil>
Dec 30 17:06:54.112551680 (601h37m53.983634715s) [noise=true , address=145.79.227.15:4875 , piece_id=NXXFDM6PISZNKGYVGIERVUI47K3QKSXRKKSWFE4MQ3HQAT3OPR4Q, seg_pos={0 0}, upload=527305]: Done uploading piece. err: <nil>
Dec 30 17:06:54.113494676 (601h37m53.984577721s) [noise=true , address=145.79.227.18:4006 , piece_id=Q63SV43LFUACDGMHDUJ74VJDCOVGESKO3BE3XPD6TKN5YDACEVVA, seg_pos={0 0}, upload=527305]: Done uploading piece. err: <nil>
Dec 30 17:06:54.390425606 (601h37m54.261508661s) [noise=true , address=77.90.31.21:4878 , piece_id=TGHI3F376RWOGTT7ZDJABFCHYF66PTTCZ5SNNVUOPPMP4DXNOJKA, seg_pos={0 0}, upload=527305]: Done uploading piece. err: <nil>
Dec 30 17:06:54.419226276 (601h37m54.290309321s) [noise=true , address=45.91.116.9:4262 , piece_id=3RXCEA5AOYR25CVEQP27AR3HTLY7NYXJKFQVK5DLR6ENSQFHZ2LA, seg_pos={0 0}, upload=527305]: Done uploading piece. err: <nil>
Dec 30 17:06:54.471359594 (601h37m54.342442629s) [noise=true , address=45.91.116.4:4940 , piece_id=3JP6EWJDZWUU5I7COBD57WCVFYTO7D33CDMZ23PA3TAE4LXGWABQ, seg_pos={0 0}, upload=527305]: Done uploading piece. err: <nil>
Dec 30 17:06:54.526497273 (601h37m54.397580318s) [noise=true , address=45.91.116.14:4192 , piece_id=67IB5J4HE7CNTANI6ZX4U7ZIKVCV4P5IBC62J6EPDFIMV4B6TXJQ, seg_pos={0 0}, upload=527305]: Done uploading piece. err: <nil>
Dec 30 17:06:54.526521899 (601h37m54.397604934s) [seg_pos={0 0}, upload=527305]: Segment reached optimal threshold of 13 pieces.
Dec 30 17:06:54.526837310 (601h37m54.397920366s) [noise=true , address=45.91.116.8:4048 , piece_id=3ZBNNFCD7B55RBVGPWP4CAGHT47RXCMLJLCDI64TTZMHC6GZPZ2A, seg_pos={0 0}, upload=527305]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF
Dec 30 17:06:54.526886303 (601h37m54.397969338s) [noise=true , address=45.91.116.8:4048 , piece_id=3ZBNNFCD7B55RBVGPWP4CAGHT47RXCMLJLCDI64TTZMHC6GZPZ2A, seg_pos={0 0}, upload=527305]: Long tail cancellation due to optimal threshold
Dec 30 17:06:54.526959072 (601h37m54.398042117s) [noise=true , address=145.79.227.44:4194 , piece_id=4ISX6KXDG35D5EGEJKLDYPLUZO5SZPRHVVAC7NRUXYUKAWPC76IQ, seg_pos={0 0}, upload=527305]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kta1F6RNDP9GJdkiUG3x1Nk6juWJ2tmSDr599DYomSq9C1BBh): protocol: expected piece hash; context canceled; EOF
Dec 30 17:06:54.526974280 (601h37m54.398057315s) [noise=true , address=145.79.227.44:4194 , piece_id=4ISX6KXDG35D5EGEJKLDYPLUZO5SZPRHVVAC7NRUXYUKAWPC76IQ, seg_pos={0 0}, upload=527305]: Long tail cancellation due to optimal threshold
Dec 30 17:06:54.527120540 (601h37m54.398203575s) [noise=true , address=77.90.31.18:4808 , piece_id=MGEWL2WO55SAQGAWVX3ZUFTSOUX3XD3VRCHQ5GMU4GAXOGGPFCXA, seg_pos={0 0}, upload=527305]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF
Dec 30 17:06:54.527125990 (601h37m54.398209035s) [noise=true , address=77.90.31.18:4808 , piece_id=MGEWL2WO55SAQGAWVX3ZUFTSOUX3XD3VRCHQ5GMU4GAXOGGPFCXA, seg_pos={0 0}, upload=527305]: Long tail cancellation due to optimal threshold
Dec 30 17:06:54.527140998 (601h37m54.398224033s) [seg_pos={0 0}, upload=527305]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 17:06:54.527218225 (601h37m54.39830126s) [upload=527305]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 17:06:54.527231421 (601h37m54.398314456s) [upload=527305]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 17:06:54.825252568 (601h37m54.696335603s) [upload=527305]: Done uploading segments...
Dec 30 17:06:54.825275332 (601h37m54.696358367s) [upload=527305]: Upload finished. err: <nil>
Duration: 30.003379104s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
7.860399775s trace:
https://jaeger.dev.storj.io/trace/4fe744e860ffef11
FAIL (2025-12-30 17:07:04 UTC)
orbiter warning: context deadline exceeded
Duration: 7.860424241s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
11.177238536s trace:
https://jaeger.dev.storj.io/trace/729389dc0e4f4bf0
FAIL (2025-12-30 17:06:59 UTC)
orbiter warning: context deadline exceeded
Duration: 11.177267713s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
18 out of 60 is failed (threshold=6) [X.X.X...XXX..X....XXX..X...XX......X.XX................X..X.] duration:
4.31950992s trace:
https://jaeger.dev.storj.io/trace/390f26c24eefb880
FAIL (2025-12-30 17:06:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.319543323s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
1m0.001633276s trace:
https://jaeger.dev.storj.io/trace/4fe744e860ffef11
FAIL (2025-12-30 17:06:48 UTC)
orbiter warning: context deadline exceeded
Duration: 1m0.001656902s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [..........X.............X.......X...........................] duration:
46.644779021s trace:
https://jaeger.dev.storj.io/trace/3f4379d3637c8e4b
FAIL (2025-12-30 17:06:38 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1WBuBdGxdPXi6VsNf4FJpn3LpWoQztM7CV6YjAW1mUDVjAUjn2, Piece ID: MS4HNXUQIITLBMEHZDPJMEH7FSLMBBGKJSIP5BY2BDUF3IE2NKIQ) context deadline exceeded; piecestore: (Node ID: 1soiTQsiMNe5BkNHwwAGykmtP9LtLxncqE7Tv728ak8pjFeshg, Piece ID: BHVQNSBU3LP35Z5WQYTRPJZNNWTDZKWYK2CMRLHSOGBJ4IXGZ5SA) context deadline exceeded; piecestore: (Node ID: 1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, Piece ID: MVRLWV5URMPV57R52J426SXY5ZWQGHOTL42GO2GPEZ4LA7B2JGCQ) context deadline exceeded
======LOGS======
5m56.099625936s) [seg_pos={0 0}, upload=526987]: Begin upload segment...
Dec 30 16:44:56.228605921 (601h15m56.099688967s) [seg_pos={0 0}, upload=526987]: Done begin upload segment.
Dec 30 16:44:56.228736641 (601h15m56.099819686s) [noise=true , address=145.79.227.34:4923 , piece_id=OB3KQEHYXE4BL73EDDQDN2LSVYJYYQQFUSFJ655BFJISLBGOH4XQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.228753061 (601h15m56.099836117s) [noise=true , address=64.49.13.26:4947 , piece_id=BHVQNSBU3LP35Z5WQYTRPJZNNWTDZKWYK2CMRLHSOGBJ4IXGZ5SA, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.229701116 (601h15m56.100784151s) [noise=true , address=45.91.116.8:4071 , piece_id=3PUQCDJAXSQGEM47BTD2B5HWWNK3PUS6AEVQ63BEC2Q2CKCB4PXQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.230399837 (601h15m56.101482882s) [noise=true , address=77.90.31.21:4899 , piece_id=DCYPDKTC4S5LG3XEBQWZT3HGFJ63T3UC3PFUDLF7OOJHNMYW4KRQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.230834926 (601h15m56.101917971s) [noise=true , address=77.90.31.19:4853 , piece_id=MS4HNXUQIITLBMEHZDPJMEH7FSLMBBGKJSIP5BY2BDUF3IE2NKIQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.232511007 (601h15m56.103594112s) [noise=true , address=145.79.227.8:4344 , piece_id=6L5DD7CQJ5JZ7IPFX2IGHNEP7M6BZU2DA5Z5GEOKGPAMVZUXQHOQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.233047428 (601h15m56.104130473s) [noise=true , address=145.79.227.31:4808 , piece_id=MW4UFSJ4XFLXK7ZLWZFWTP2N4QQUS24WP3O44DJWT7AK2WCWAT3Q, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.233557500 (601h15m56.104640545s) [noise=true , address=145.79.227.33:4903 , piece_id=THSFTSO2Y4QVWIGNUOU4AKFGTSQWQ6FLNIIDISIVNRF7XZZWWPCQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.234065937 (601h15m56.105148983s) [noise=true , address=64.49.13.137:4493 , piece_id=QZSPIHMAGKSHSAJYJ6BR26EMTCWFAEPOPC4O65YN6SL6222H4HWQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.234513860 (601h15m56.105596896s) [noise=true , address=64.49.13.27:4965 , piece_id=AQTJD7BXKPMIAGEOGJCY5Y64IVU2SHCZEFJL2NLQAHVX5HR33N7Q, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.236125519 (601h15m56.107208554s) [noise=true , address=77.90.31.11:4567 , piece_id=NMCHY46TZ6W6EPVLVVCGQXHFERHBLSFB3J55AD32KQSKOVXABO7A, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.237179756 (601h15m56.108262791s) [noise=true , address=145.79.227.40:4064 , piece_id=SEYSDBZHX6SHMFZBGY3XDIYEOTWFCNGG2KNNTCIXYQRZP7FZOVJQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.238439144 (601h15m56.109522189s) [noise=true , address=145.79.227.30:4790 , piece_id=5VRI4FYK7J54TCOY7DX6TCQ3634N3TATGDZUZDWRBBLMXCGRVR3Q, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.239316585 (601h15m56.11039962s) [noise=true , address=145.79.227.44:4212 , piece_id=ZV3XMTVGJR2D5X4EEGGDTGGGXSQI4QNCJZCI3GPR6RTD7JEQWLGA, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.242138468 (601h15m56.113221503s) [noise=true , address=45.91.116.11:4131 , piece_id=YAFIDLBM762GWTNKDW62FMNLE4YHCNYW2WU3R4AMO5NELMVOHHHQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.243965547 (601h15m56.115048582s) [noise=true , address=77.90.31.18:4829 , piece_id=MVRLWV5URMPV57R52J426SXY5ZWQGHOTL42GO2GPEZ4LA7B2JGCQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.250805150 (601h15m56.121888185s) [noise=true , address=5.231.73.4:4033 , piece_id=CCH6AH47YBQPGPF4MPZU2NPT7HPF4D3KT3ZVDITRYF7UW5JUKWDQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.254973396 (601h15m56.126056431s) [noise=true , address=145.79.227.49:4341 , piece_id=6GKQQIBOECLDEEDZNO5AYKX7XMXZYUA6WNGKD3FQSF2Q4O6A4MQA, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:57.691193696 (601h15m57.562276731s) [upload=526987]: Next returned nil segment
Dec 30 16:44:57.691206460 (601h15m57.562289495s) [upload=526987]: Waiting for error group managing segments...
Dec 30 16:44:58.085111563 (601h15m57.956194598s) [noise=true , address=64.49.13.137:4493 , piece_id=QZSPIHMAGKSHSAJYJ6BR26EMTCWFAEPOPC4O65YN6SL6222H4HWQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.086614434 (601h15m57.957697479s) [noise=true , address=64.49.13.27:4965 , piece_id=AQTJD7BXKPMIAGEOGJCY5Y64IVU2SHCZEFJL2NLQAHVX5HR33N7Q, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.119421281 (601h15m57.990504316s) [noise=true , address=145.79.227.8:4344 , piece_id=6L5DD7CQJ5JZ7IPFX2IGHNEP7M6BZU2DA5Z5GEOKGPAMVZUXQHOQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.125317667 (601h15m57.996400702s) [noise=true , address=145.79.227.31:4808 , piece_id=MW4UFSJ4XFLXK7ZLWZFWTP2N4QQUS24WP3O44DJWT7AK2WCWAT3Q, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.262371457 (601h15m58.133454502s) [noise=true , address=64.49.13.26:4947 , piece_id=BHVQNSBU3LP35Z5WQYTRPJZNNWTDZKWYK2CMRLHSOGBJ4IXGZ5SA, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.284980216 (601h15m58.156063251s) [noise=true , address=77.90.31.19:4853 , piece_id=MS4HNXUQIITLBMEHZDPJMEH7FSLMBBGKJSIP5BY2BDUF3IE2NKIQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.387549123 (601h15m58.258632158s) [noise=true , address=77.90.31.18:4829 , piece_id=MVRLWV5URMPV57R52J426SXY5ZWQGHOTL42GO2GPEZ4LA7B2JGCQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.394280961 (601h15m58.265364006s) [noise=true , address=5.231.73.4:4033 , piece_id=CCH6AH47YBQPGPF4MPZU2NPT7HPF4D3KT3ZVDITRYF7UW5JUKWDQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.510328564 (601h15m58.381411609s) [noise=true , address=45.91.116.8:4071 , piece_id=3PUQCDJAXSQGEM47BTD2B5HWWNK3PUS6AEVQ63BEC2Q2CKCB4PXQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.545484694 (601h15m58.416567739s) [noise=true , address=45.91.116.11:4131 , piece_id=YAFIDLBM762GWTNKDW62FMNLE4YHCNYW2WU3R4AMO5NELMVOHHHQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.710409562 (601h15m58.581492607s) [noise=true , address=77.90.31.21:4899 , piece_id=DCYPDKTC4S5LG3XEBQWZT3HGFJ63T3UC3PFUDLF7OOJHNMYW4KRQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.981893937 (601h15m58.852976982s) [noise=true , address=145.79.227.44:4212 , piece_id=ZV3XMTVGJR2D5X4EEGGDTGGGXSQI4QNCJZCI3GPR6RTD7JEQWLGA, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.995663946 (601h15m58.866746991s) [noise=true , address=145.79.227.49:4341 , piece_id=6GKQQIBOECLDEEDZNO5AYKX7XMXZYUA6WNGKD3FQSF2Q4O6A4MQA, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:59.009469091 (601h15m58.880552136s) [noise=true , address=145.79.227.40:4064 , piece_id=SEYSDBZHX6SHMFZBGY3XDIYEOTWFCNGG2KNNTCIXYQRZP7FZOVJQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:59.009489950 (601h15m58.880572986s) [seg_pos={0 0}, upload=526987]: Segment reached optimal threshold of 14 pieces.
Dec 30 16:44:59.009662329 (601h15m58.880745374s) [noise=true , address=145.79.227.34:4923 , piece_id=OB3KQEHYXE4BL73EDDQDN2LSVYJYYQQFUSFJ655BFJISLBGOH4XQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VXAxRGiJQor24W7DYSeUQUu1QF7EcX6BQYgfUheACRs2quFFu): protocol: expected piece hash; context canceled; EOF
Dec 30 16:44:59.009668390 (601h15m58.880751425s) [noise=true , address=145.79.227.34:4923 , piece_id=OB3KQEHYXE4BL73EDDQDN2LSVYJYYQQFUSFJ655BFJISLBGOH4XQ, seg_pos={0 0}, upload=526987]: Long tail cancellation due to optimal threshold
Dec 30 16:44:59.009857671 (601h15m58.880940706s) [noise=true , address=145.79.227.33:4903 , piece_id=THSFTSO2Y4QVWIGNUOU4AKFGTSQWQ6FLNIIDISIVNRF7XZZWWPCQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:59.009921232 (601h15m58.881004267s) [noise=true , address=145.79.227.30:4790 , piece_id=5VRI4FYK7J54TCOY7DX6TCQ3634N3TATGDZUZDWRBBLMXCGRVR3Q, seg_pos={0 0}, upload=526987]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kuSMof8EbvsaMCdVxKWjgTT5TfLJnTAeHUf8DKNC3UGQn5q7T): protocol: expected piece hash; context canceled; EOF
Dec 30 16:44:59.009927314 (601h15m58.881010359s) [noise=true , address=145.79.227.30:4790 , piece_id=5VRI4FYK7J54TCOY7DX6TCQ3634N3TATGDZUZDWRBBLMXCGRVR3Q, seg_pos={0 0}, upload=526987]: Long tail cancellation due to optimal threshold
Dec 30 16:44:59.010092589 (601h15m58.881175624s) [noise=true , address=77.90.31.11:4567 , piece_id=NMCHY46TZ6W6EPVLVVCGQXHFERHBLSFB3J55AD32KQSKOVXABO7A, seg_pos={0 0}, upload=526987]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HJVmEuxjrwWCcdwxWYFC5fTULds1kiRH82DiRWipWhRT1Qwdk): protocol: expected piece hash; context canceled; EOF
Dec 30 16:44:59.010101946 (601h15m58.881184981s) [noise=true , address=77.90.31.11:4567 , piece_id=NMCHY46TZ6W6EPVLVVCGQXHFERHBLSFB3J55AD32KQSKOVXABO7A, seg_pos={0 0}, upload=526987]: Long tail cancellation due to optimal threshold
Dec 30 16:44:59.010125410 (601h15m58.881208445s) [seg_pos={0 0}, upload=526987]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil>
Dec 30 16:44:59.010204091 (601h15m58.881287136s) [upload=526987]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 16:44:59.010211274 (601h15m58.881294339s) [upload=526987]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 16:44:59.309660956 (601h15m59.180744001s) [upload=526987]: Done uploading segments...
Dec 30 16:44:59.309679662 (601h15m59.180762697s) [upload=526987]: Upload finished. err: <nil>
Duration: 46.64480946s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
18 out of 60 is failed (threshold=6) [.X.X.X...XXX..X....XXX..X...XX......X.XX................X..X] duration:
5.001440904s trace:
https://jaeger.dev.storj.io/trace/390f26c24eefb880
FAIL (2025-12-30 17:06:29 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001458629s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.001188252s trace:
https://jaeger.dev.storj.io/trace/729389dc0e4f4bf0
FAIL (2025-12-30 17:06:25 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001205635s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
23 out of 60 is failed (threshold=3) [X........X..X....X.XXX..XX.X.X..XXX.XX..X.X.......X.X..XX.X.] duration:
6.102422242s trace:
https://jaeger.dev.storj.io/trace/298a64c694161836
FAIL (2025-12-30 17:06:16 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: MNLXFGXTLJFA7BZE3AIHNFZ2BPXIV6LWXHFDCKPNIM5AKMPF2UBQ) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: PLFP2UW2BYVMNNAMGOBBGKKBFCIL7UNKY2JFXXLKIP6C57FTZAYA) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: KTKTSMB5TNATYNSI7KDDTO3Q3Z5DZE6OWW27EFR4D22OUIT64C3A) context deadline exceeded
======LOGS======
Dec 30 17:05:12.410840911 (601h36m12.281923946s) [upload=527283]: Starting upload
Dec 30 17:05:12.410924751 (601h36m12.282007806s) [upload=527283]: Done starting upload
Dec 30 17:05:12.411156131 (601h36m12.282239176s) [upload=527283]: Uploading segments...
Dec 30 17:05:12.411268605 (601h36m12.28235164s) [upload=527283]: Got next segment. Inline: false
Dec 30 17:05:12.411327908 (601h36m12.282410944s) [upload=527283]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 17:05:12.411333930 (601h36m12.282416975s) [upload=527283]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 17:05:12.658541012 (601h36m12.529624057s) [seg_pos={0 0}, upload=527283]: Begin upload segment...
Dec 30 17:05:12.658588143 (601h36m12.529671178s) [seg_pos={0 0}, upload=527283]: Done begin upload segment.
Dec 30 17:05:12.658613811 (601h36m12.529696846s) [noise=true , address=145.79.227.45:4231 , piece_id=WFMMVKKBMAVX73632NG323WYCSAUSCVI6U6D25N7KYMLAYEQXPJA, seg_pos={0 0}, upload=527283]: Uploading piece...
Dec 30 17:05:12.658643578 (601h36m12.529726623s) [noise=true , address=145.79.227.52:4447 , piece_id=3EYFJDDFXTBBD7HASTJHQU3H3AC7AU334DY2YBE5JCDDMIOC44YQ, seg_pos={0 0}, upload=527283]: Uploading piece...
Dec 30 17:05:12.659356024 (601h36m12.530439069s) [noise=true , address=77.90.31.10:4519 , piece_id=KTKTSMB5TNATYNSI7KDDTO3Q3Z5DZE6OWW27EFR4D22OUIT64C3A, seg_pos={0 0}, upload=527283]: Uploading piece...
Dec 30 17:05:12.659939074 (601h36m12.531022119s) [noise=true , address=145.79.227.41:4087 , piece_id=UG6BHBAYOUMVCYXFKA5EMXED3O2NFJQYEPXTFGU7EIK4CZMGVOMA, seg_pos={0 0}, upload=527283]: Uploading piece...
Dec 30 17:05:12.660324378 (601h36m12.531407433s) [noise=true , address=145.79.227.15:4874 , piece_id=C5R5VKKJTGSH643ZWOG3M7OAFY2MYYHG6SWW6HHJZH2VKZ4TEDAA, seg_pos={0 0}, upload=527283]: Uploading piece...
Dec 30 17:05:12.660920904 (601h36m12.532003959s) [noise=true , address=145.79.227.46:4603 , piece_id=OYJE4RUDC26MXXB3EPBU77BN75QFXHNQXA6QVZWER36XLVF6CKJA, seg_pos={0 0}, upload=527283]: Uploading piece...
Dec 30 17:05:12.661503613 (601h36m12.532586659s) [noise=true , address=145.79.227.23:4547 , piece_id=Z2WQ6BVZENTGFEKNSHSHMKKV4NHXHURT4P7DRR32XKSRK7RHIHBA, seg_pos={0 0}, upload=527283]: Uploading piece...
Dec 30 17:05:12.662368400 (601h36m12.533451445s) [noise=true , address=145.79.227.51:4410 , piece_id=LZIQAELYI5VYX3FALEDFOLIKBTAEHI2VGK2UAIGO3BPJL4DNNQWA, seg_pos={0 0}, upload=527283]: Uploading piece...
Dec 30 17:05:12.662834277 (601h36m12.533917322s) [noise=true , address=145.79.227.38:4979 , piece_id=NWQJAKTL4QLH7NO7NBDZ3HOAD5ZNWTEVZEU27RL4ZFZKQUW6YBCQ, seg_pos={0 0}, upload=527283]: Uploading piece...
Dec 30 17:05:12.663263394 (601h36m12.534346439s) [noise=true , address=145.79.227.31:4797 , piece_id=6NNKOTUYQI6SVUR42L3ZHWMSPS3OW4B2RZRJMBNCIL55DWUUGB6A, seg_pos={0 0}, upload=527283]: Uploading piece...
Dec 30 17:05:12.663335943 (601h36m12.534418988s) [noise=true , address=145.79.227.9:4366 , piece_id=RZ3Y36UR3KPUD63TY4IMBU6W52CYW5UDCAODT3AH45BB3N2PDD7A, seg_pos={0 0}, upload=527283]: Uploading piece...
Dec 30 17:05:12.663805146 (601h36m12.534888191s) [noise=true , address=77.90.31.17:4770 , piece_id=IQP5SME5SDW3GYTCDFEKT7EMR72OVWFCAZA36DK6PFKL2P2I3OYQ, seg_pos={0 0}, upload=527283]: Uploading piece...
Dec 30 17:05:12.663983175 (601h36m12.53506622s) [noise=true , address=45.91.116.8:4048 , piece_id=MNLXFGXTLJFA7BZE3AIHNFZ2BPXIV6LWXHFDCKPNIM5AKMPF2UBQ, seg_pos={0 0}, upload=527283]: Uploading piece...
Dec 30 17:05:12.664297804 (601h36m12.535380849s) [noise=true , address=145.79.227.8:4331 , piece_id=L6BUZ4SELJ3OJTP23SJNJRYZLVOKIR3HQHNHDKRSCQKUMOYQXZ4Q, seg_pos={0 0}, upload=527283]: Uploading piece...
Dec 30 17:05:12.664494859 (601h36m12.535577894s) [noise=true , address=45.91.116.5:4298 , piece_id=2FBLSG2SXBV3VL2IUPFSH6ENUC3DSXCM2DUEI4YR3WEZHE5G6GOA, seg_pos={0 0}, upload=527283]: Uploading piece...
Dec 30 17:05:12.664725388 (601h36m12.535808433s) [noise=true , address=45.91.116.13:4156 , piece_id=PLFP2UW2BYVMNNAMGOBBGKKBFCIL7UNKY2JFXXLKIP6C57FTZAYA, seg_pos={0 0}, upload=527283]: Uploading piece...
Dec 30 17:05:14.359906439 (601h36m14.230989484s) [upload=527283]: Next returned nil segment
Dec 30 17:05:14.359919534 (601h36m14.231002569s) [upload=527283]: Waiting for error group managing segments...
Dec 30 17:05:14.632538183 (601h36m14.503621228s) [noise=true , address=145.79.227.15:4874 , piece_id=C5R5VKKJTGSH643ZWOG3M7OAFY2MYYHG6SWW6HHJZH2VKZ4TEDAA, seg_pos={0 0}, upload=527283]: Done uploading piece. err: <nil>
Dec 30 17:05:14.652301455 (601h36m14.52338449s) [noise=true , address=145.79.227.9:4366 , piece_id=RZ3Y36UR3KPUD63TY4IMBU6W52CYW5UDCAODT3AH45BB3N2PDD7A, seg_pos={0 0}, upload=527283]: Done uploading piece. err: <nil>
Dec 30 17:05:14.653584418 (601h36m14.524667453s) [noise=true , address=145.79.227.31:4797 , piece_id=6NNKOTUYQI6SVUR42L3ZHWMSPS3OW4B2RZRJMBNCIL55DWUUGB6A, seg_pos={0 0}, upload=527283]: Done uploading piece. err: <nil>
Dec 30 17:05:14.653768679 (601h36m14.524851724s) [noise=true , address=145.79.227.38:4979 , piece_id=NWQJAKTL4QLH7NO7NBDZ3HOAD5ZNWTEVZEU27RL4ZFZKQUW6YBCQ, seg_pos={0 0}, upload=527283]: Done uploading piece. err: <nil>
Dec 30 17:05:14.664563523 (601h36m14.535646568s) [noise=true , address=145.79.227.41:4087 , piece_id=UG6BHBAYOUMVCYXFKA5EMXED3O2NFJQYEPXTFGU7EIK4CZMGVOMA, seg_pos={0 0}, upload=527283]: Done uploading piece. err: <nil>
Dec 30 17:05:14.667513310 (601h36m14.538596345s) [noise=true , address=145.79.227.46:4603 , piece_id=OYJE4RUDC26MXXB3EPBU77BN75QFXHNQXA6QVZWER36XLVF6CKJA, seg_pos={0 0}, upload=527283]: Done uploading piece. err: <nil>
Dec 30 17:05:14.671864997 (601h36m14.542948032s) [noise=true , address=145.79.227.45:4231 , piece_id=WFMMVKKBMAVX73632NG323WYCSAUSCVI6U6D25N7KYMLAYEQXPJA, seg_pos={0 0}, upload=527283]: Done uploading piece. err: <nil>
Dec 30 17:05:14.686363423 (601h36m14.557446458s) [noise=true , address=145.79.227.23:4547 , piece_id=Z2WQ6BVZENTGFEKNSHSHMKKV4NHXHURT4P7DRR32XKSRK7RHIHBA, seg_pos={0 0}, upload=527283]: Done uploading piece. err: <nil>
Dec 30 17:05:14.690210268 (601h36m14.561293323s) [noise=true , address=145.79.227.52:4447 , piece_id=3EYFJDDFXTBBD7HASTJHQU3H3AC7AU334DY2YBE5JCDDMIOC44YQ, seg_pos={0 0}, upload=527283]: Done uploading piece. err: <nil>
Dec 30 17:05:14.717188998 (601h36m14.588272043s) [noise=true , address=145.79.227.51:4410 , piece_id=LZIQAELYI5VYX3FALEDFOLIKBTAEHI2VGK2UAIGO3BPJL4DNNQWA, seg_pos={0 0}, upload=527283]: Done uploading piece. err: <nil>
Dec 30 17:05:14.730613370 (601h36m14.601696405s) [noise=true , address=145.79.227.8:4331 , piece_id=L6BUZ4SELJ3OJTP23SJNJRYZLVOKIR3HQHNHDKRSCQKUMOYQXZ4Q, seg_pos={0 0}, upload=527283]: Done uploading piece. err: <nil>
Dec 30 17:05:14.826629066 (601h36m14.697712101s) [noise=true , address=77.90.31.10:4519 , piece_id=KTKTSMB5TNATYNSI7KDDTO3Q3Z5DZE6OWW27EFR4D22OUIT64C3A, seg_pos={0 0}, upload=527283]: Done uploading piece. err: <nil>
Dec 30 17:05:14.984672639 (601h36m14.855755675s) [noise=true , address=45.91.116.8:4048 , piece_id=MNLXFGXTLJFA7BZE3AIHNFZ2BPXIV6LWXHFDCKPNIM5AKMPF2UBQ, seg_pos={0 0}, upload=527283]: Done uploading piece. err: <nil>
Dec 30 17:05:14.984697287 (601h36m14.855780342s) [seg_pos={0 0}, upload=527283]: Segment reached optimal threshold of 13 pieces.
Dec 30 17:05:14.984928226 (601h36m14.856011271s) [noise=true , address=45.91.116.13:4156 , piece_id=PLFP2UW2BYVMNNAMGOBBGKKBFCIL7UNKY2JFXXLKIP6C57FTZAYA, seg_pos={0 0}, upload=527283]: Done uploading piece. err: <nil>
Dec 30 17:05:14.984959015 (601h36m14.85604206s) [noise=true , address=45.91.116.5:4298 , piece_id=2FBLSG2SXBV3VL2IUPFSH6ENUC3DSXCM2DUEI4YR3WEZHE5G6GOA, seg_pos={0 0}, upload=527283]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF
Dec 30 17:05:14.984967370 (601h36m14.856050435s) [noise=true , address=45.91.116.5:4298 , piece_id=2FBLSG2SXBV3VL2IUPFSH6ENUC3DSXCM2DUEI4YR3WEZHE5G6GOA, seg_pos={0 0}, upload=527283]: Long tail cancellation due to optimal threshold
Dec 30 17:05:14.985304963 (601h36m14.856387998s) [noise=true , address=77.90.31.17:4770 , piece_id=IQP5SME5SDW3GYTCDFEKT7EMR72OVWFCAZA36DK6PFKL2P2I3OYQ, seg_pos={0 0}, upload=527283]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: read tcp 188.166.225.170:50432->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 30 17:05:14.985314652 (601h36m14.856397697s) [noise=true , address=77.90.31.17:4770 , piece_id=IQP5SME5SDW3GYTCDFEKT7EMR72OVWFCAZA36DK6PFKL2P2I3OYQ, seg_pos={0 0}, upload=527283]: Long tail cancellation due to optimal threshold
Dec 30 17:05:14.985334901 (601h36m14.856417936s) [seg_pos={0 0}, upload=527283]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil>
Dec 30 17:05:14.985421695 (601h36m14.85650473s) [upload=527283]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 17:05:14.985430974 (601h36m14.856514029s) [upload=527283]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 17:05:15.277727682 (601h36m15.148810737s) [upload=527283]: Done uploading segments...
Dec 30 17:05:15.277750326 (601h36m15.148833361s) [upload=527283]: Upload finished. err: <nil>
Duration: 6.102452409s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
24 out of 60 is failed (threshold=3) [XX........X..X....X.XXX..XX.X.X..XXX.XX..X.X.......X.X..XX.X] duration:
30.003887736s trace:
https://jaeger.dev.storj.io/trace/298a64c694161836
FAIL (2025-12-30 17:05:41 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: MNLXFGXTLJFA7BZE3AIHNFZ2BPXIV6LWXHFDCKPNIM5AKMPF2UBQ) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: PLFP2UW2BYVMNNAMGOBBGKKBFCIL7UNKY2JFXXLKIP6C57FTZAYA) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: KTKTSMB5TNATYNSI7KDDTO3Q3Z5DZE6OWW27EFR4D22OUIT64C3A) context deadline exceeded
======LOGS======
Dec 30 17:05:12.410840911 (601h36m12.281923946s) [upload=527283]: Starting upload
Dec 30 17:05:12.410924751 (601h36m12.282007806s) [upload=527283]: Done starting upload
Dec 30 17:05:12.411156131 (601h36m12.282239176s) [upload=527283]: Uploading segments...
Dec 30 17:05:12.411268605 (601h36m12.28235164s) [upload=527283]: Got next segment. Inline: false
Dec 30 17:05:12.411327908 (601h36m12.282410944s) [upload=527283]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 17:05:12.411333930 (601h36m12.282416975s) [upload=527283]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 17:05:12.658541012 (601h36m12.529624057s) [seg_pos={0 0}, upload=527283]: Begin upload segment...
Dec 30 17:05:12.658588143 (601h36m12.529671178s) [seg_pos={0 0}, upload=527283]: Done begin upload segment.
Dec 30 17:05:12.658613811 (601h36m12.529696846s) [noise=true , address=145.79.227.45:4231 , piece_id=WFMMVKKBMAVX73632NG323WYCSAUSCVI6U6D25N7KYMLAYEQXPJA, seg_pos={0 0}, upload=527283]: Uploading piece...
Dec 30 17:05:12.658643578 (601h36m12.529726623s) [noise=true , address=145.79.227.52:4447 , piece_id=3EYFJDDFXTBBD7HASTJHQU3H3AC7AU334DY2YBE5JCDDMIOC44YQ, seg_pos={0 0}, upload=527283]: Uploading piece...
Dec 30 17:05:12.659356024 (601h36m12.530439069s) [noise=true , address=77.90.31.10:4519 , piece_id=KTKTSMB5TNATYNSI7KDDTO3Q3Z5DZE6OWW27EFR4D22OUIT64C3A, seg_pos={0 0}, upload=527283]: Uploading piece...
Dec 30 17:05:12.659939074 (601h36m12.531022119s) [noise=true , address=145.79.227.41:4087 , piece_id=UG6BHBAYOUMVCYXFKA5EMXED3O2NFJQYEPXTFGU7EIK4CZMGVOMA, seg_pos={0 0}, upload=527283]: Uploading piece...
Dec 30 17:05:12.660324378 (601h36m12.531407433s) [noise=true , address=145.79.227.15:4874 , piece_id=C5R5VKKJTGSH643ZWOG3M7OAFY2MYYHG6SWW6HHJZH2VKZ4TEDAA, seg_pos={0 0}, upload=527283]: Uploading piece...
Dec 30 17:05:12.660920904 (601h36m12.532003959s) [noise=true , address=145.79.227.46:4603 , piece_id=OYJE4RUDC26MXXB3EPBU77BN75QFXHNQXA6QVZWER36XLVF6CKJA, seg_pos={0 0}, upload=527283]: Uploading piece...
Dec 30 17:05:12.661503613 (601h36m12.532586659s) [noise=true , address=145.79.227.23:4547 , piece_id=Z2WQ6BVZENTGFEKNSHSHMKKV4NHXHURT4P7DRR32XKSRK7RHIHBA, seg_pos={0 0}, upload=527283]: Uploading piece...
Dec 30 17:05:12.662368400 (601h36m12.533451445s) [noise=true , address=145.79.227.51:4410 , piece_id=LZIQAELYI5VYX3FALEDFOLIKBTAEHI2VGK2UAIGO3BPJL4DNNQWA, seg_pos={0 0}, upload=527283]: Uploading piece...
Dec 30 17:05:12.662834277 (601h36m12.533917322s) [noise=true , address=145.79.227.38:4979 , piece_id=NWQJAKTL4QLH7NO7NBDZ3HOAD5ZNWTEVZEU27RL4ZFZKQUW6YBCQ, seg_pos={0 0}, upload=527283]: Uploading piece...
Dec 30 17:05:12.663263394 (601h36m12.534346439s) [noise=true , address=145.79.227.31:4797 , piece_id=6NNKOTUYQI6SVUR42L3ZHWMSPS3OW4B2RZRJMBNCIL55DWUUGB6A, seg_pos={0 0}, upload=527283]: Uploading piece...
Dec 30 17:05:12.663335943 (601h36m12.534418988s) [noise=true , address=145.79.227.9:4366 , piece_id=RZ3Y36UR3KPUD63TY4IMBU6W52CYW5UDCAODT3AH45BB3N2PDD7A, seg_pos={0 0}, upload=527283]: Uploading piece...
Dec 30 17:05:12.663805146 (601h36m12.534888191s) [noise=true , address=77.90.31.17:4770 , piece_id=IQP5SME5SDW3GYTCDFEKT7EMR72OVWFCAZA36DK6PFKL2P2I3OYQ, seg_pos={0 0}, upload=527283]: Uploading piece...
Dec 30 17:05:12.663983175 (601h36m12.53506622s) [noise=true , address=45.91.116.8:4048 , piece_id=MNLXFGXTLJFA7BZE3AIHNFZ2BPXIV6LWXHFDCKPNIM5AKMPF2UBQ, seg_pos={0 0}, upload=527283]: Uploading piece...
Dec 30 17:05:12.664297804 (601h36m12.535380849s) [noise=true , address=145.79.227.8:4331 , piece_id=L6BUZ4SELJ3OJTP23SJNJRYZLVOKIR3HQHNHDKRSCQKUMOYQXZ4Q, seg_pos={0 0}, upload=527283]: Uploading piece...
Dec 30 17:05:12.664494859 (601h36m12.535577894s) [noise=true , address=45.91.116.5:4298 , piece_id=2FBLSG2SXBV3VL2IUPFSH6ENUC3DSXCM2DUEI4YR3WEZHE5G6GOA, seg_pos={0 0}, upload=527283]: Uploading piece...
Dec 30 17:05:12.664725388 (601h36m12.535808433s) [noise=true , address=45.91.116.13:4156 , piece_id=PLFP2UW2BYVMNNAMGOBBGKKBFCIL7UNKY2JFXXLKIP6C57FTZAYA, seg_pos={0 0}, upload=527283]: Uploading piece...
Dec 30 17:05:14.359906439 (601h36m14.230989484s) [upload=527283]: Next returned nil segment
Dec 30 17:05:14.359919534 (601h36m14.231002569s) [upload=527283]: Waiting for error group managing segments...
Dec 30 17:05:14.632538183 (601h36m14.503621228s) [noise=true , address=145.79.227.15:4874 , piece_id=C5R5VKKJTGSH643ZWOG3M7OAFY2MYYHG6SWW6HHJZH2VKZ4TEDAA, seg_pos={0 0}, upload=527283]: Done uploading piece. err: <nil>
Dec 30 17:05:14.652301455 (601h36m14.52338449s) [noise=true , address=145.79.227.9:4366 , piece_id=RZ3Y36UR3KPUD63TY4IMBU6W52CYW5UDCAODT3AH45BB3N2PDD7A, seg_pos={0 0}, upload=527283]: Done uploading piece. err: <nil>
Dec 30 17:05:14.653584418 (601h36m14.524667453s) [noise=true , address=145.79.227.31:4797 , piece_id=6NNKOTUYQI6SVUR42L3ZHWMSPS3OW4B2RZRJMBNCIL55DWUUGB6A, seg_pos={0 0}, upload=527283]: Done uploading piece. err: <nil>
Dec 30 17:05:14.653768679 (601h36m14.524851724s) [noise=true , address=145.79.227.38:4979 , piece_id=NWQJAKTL4QLH7NO7NBDZ3HOAD5ZNWTEVZEU27RL4ZFZKQUW6YBCQ, seg_pos={0 0}, upload=527283]: Done uploading piece. err: <nil>
Dec 30 17:05:14.664563523 (601h36m14.535646568s) [noise=true , address=145.79.227.41:4087 , piece_id=UG6BHBAYOUMVCYXFKA5EMXED3O2NFJQYEPXTFGU7EIK4CZMGVOMA, seg_pos={0 0}, upload=527283]: Done uploading piece. err: <nil>
Dec 30 17:05:14.667513310 (601h36m14.538596345s) [noise=true , address=145.79.227.46:4603 , piece_id=OYJE4RUDC26MXXB3EPBU77BN75QFXHNQXA6QVZWER36XLVF6CKJA, seg_pos={0 0}, upload=527283]: Done uploading piece. err: <nil>
Dec 30 17:05:14.671864997 (601h36m14.542948032s) [noise=true , address=145.79.227.45:4231 , piece_id=WFMMVKKBMAVX73632NG323WYCSAUSCVI6U6D25N7KYMLAYEQXPJA, seg_pos={0 0}, upload=527283]: Done uploading piece. err: <nil>
Dec 30 17:05:14.686363423 (601h36m14.557446458s) [noise=true , address=145.79.227.23:4547 , piece_id=Z2WQ6BVZENTGFEKNSHSHMKKV4NHXHURT4P7DRR32XKSRK7RHIHBA, seg_pos={0 0}, upload=527283]: Done uploading piece. err: <nil>
Dec 30 17:05:14.690210268 (601h36m14.561293323s) [noise=true , address=145.79.227.52:4447 , piece_id=3EYFJDDFXTBBD7HASTJHQU3H3AC7AU334DY2YBE5JCDDMIOC44YQ, seg_pos={0 0}, upload=527283]: Done uploading piece. err: <nil>
Dec 30 17:05:14.717188998 (601h36m14.588272043s) [noise=true , address=145.79.227.51:4410 , piece_id=LZIQAELYI5VYX3FALEDFOLIKBTAEHI2VGK2UAIGO3BPJL4DNNQWA, seg_pos={0 0}, upload=527283]: Done uploading piece. err: <nil>
Dec 30 17:05:14.730613370 (601h36m14.601696405s) [noise=true , address=145.79.227.8:4331 , piece_id=L6BUZ4SELJ3OJTP23SJNJRYZLVOKIR3HQHNHDKRSCQKUMOYQXZ4Q, seg_pos={0 0}, upload=527283]: Done uploading piece. err: <nil>
Dec 30 17:05:14.826629066 (601h36m14.697712101s) [noise=true , address=77.90.31.10:4519 , piece_id=KTKTSMB5TNATYNSI7KDDTO3Q3Z5DZE6OWW27EFR4D22OUIT64C3A, seg_pos={0 0}, upload=527283]: Done uploading piece. err: <nil>
Dec 30 17:05:14.984672639 (601h36m14.855755675s) [noise=true , address=45.91.116.8:4048 , piece_id=MNLXFGXTLJFA7BZE3AIHNFZ2BPXIV6LWXHFDCKPNIM5AKMPF2UBQ, seg_pos={0 0}, upload=527283]: Done uploading piece. err: <nil>
Dec 30 17:05:14.984697287 (601h36m14.855780342s) [seg_pos={0 0}, upload=527283]: Segment reached optimal threshold of 13 pieces.
Dec 30 17:05:14.984928226 (601h36m14.856011271s) [noise=true , address=45.91.116.13:4156 , piece_id=PLFP2UW2BYVMNNAMGOBBGKKBFCIL7UNKY2JFXXLKIP6C57FTZAYA, seg_pos={0 0}, upload=527283]: Done uploading piece. err: <nil>
Dec 30 17:05:14.984959015 (601h36m14.85604206s) [noise=true , address=45.91.116.5:4298 , piece_id=2FBLSG2SXBV3VL2IUPFSH6ENUC3DSXCM2DUEI4YR3WEZHE5G6GOA, seg_pos={0 0}, upload=527283]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF
Dec 30 17:05:14.984967370 (601h36m14.856050435s) [noise=true , address=45.91.116.5:4298 , piece_id=2FBLSG2SXBV3VL2IUPFSH6ENUC3DSXCM2DUEI4YR3WEZHE5G6GOA, seg_pos={0 0}, upload=527283]: Long tail cancellation due to optimal threshold
Dec 30 17:05:14.985304963 (601h36m14.856387998s) [noise=true , address=77.90.31.17:4770 , piece_id=IQP5SME5SDW3GYTCDFEKT7EMR72OVWFCAZA36DK6PFKL2P2I3OYQ, seg_pos={0 0}, upload=527283]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: read tcp 188.166.225.170:50432->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 30 17:05:14.985314652 (601h36m14.856397697s) [noise=true , address=77.90.31.17:4770 , piece_id=IQP5SME5SDW3GYTCDFEKT7EMR72OVWFCAZA36DK6PFKL2P2I3OYQ, seg_pos={0 0}, upload=527283]: Long tail cancellation due to optimal threshold
Dec 30 17:05:14.985334901 (601h36m14.856417936s) [seg_pos={0 0}, upload=527283]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil>
Dec 30 17:05:14.985421695 (601h36m14.85650473s) [upload=527283]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 17:05:14.985430974 (601h36m14.856514029s) [upload=527283]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 17:05:15.277727682 (601h36m15.148810737s) [upload=527283]: Done uploading segments...
Dec 30 17:05:15.277750326 (601h36m15.148833361s) [upload=527283]: Upload finished. err: <nil>
Duration: 30.003899358s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
18 out of 60 is failed (threshold=6) [X.X.X.X...XXX..X....XXX..X...XX......X.XX................X..] duration:
3.636735096s trace:
https://jaeger.dev.storj.io/trace/16a0763f8dcf089e
FAIL (2025-12-30 17:05:34 UTC)
orbiter warning: context deadline exceeded
Duration: 3.636760685s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [...........X.............X.......X..........................] duration:
23.855624359s trace:
https://jaeger.dev.storj.io/trace/3f4379d3637c8e4b
FAIL (2025-12-30 17:05:26 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1WBuBdGxdPXi6VsNf4FJpn3LpWoQztM7CV6YjAW1mUDVjAUjn2, Piece ID: MS4HNXUQIITLBMEHZDPJMEH7FSLMBBGKJSIP5BY2BDUF3IE2NKIQ) context deadline exceeded; piecestore: (Node ID: 1soiTQsiMNe5BkNHwwAGykmtP9LtLxncqE7Tv728ak8pjFeshg, Piece ID: BHVQNSBU3LP35Z5WQYTRPJZNNWTDZKWYK2CMRLHSOGBJ4IXGZ5SA) context deadline exceeded; piecestore: (Node ID: 1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, Piece ID: MVRLWV5URMPV57R52J426SXY5ZWQGHOTL42GO2GPEZ4LA7B2JGCQ) context deadline exceeded
======LOGS======
5m56.099625936s) [seg_pos={0 0}, upload=526987]: Begin upload segment...
Dec 30 16:44:56.228605921 (601h15m56.099688967s) [seg_pos={0 0}, upload=526987]: Done begin upload segment.
Dec 30 16:44:56.228736641 (601h15m56.099819686s) [noise=true , address=145.79.227.34:4923 , piece_id=OB3KQEHYXE4BL73EDDQDN2LSVYJYYQQFUSFJ655BFJISLBGOH4XQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.228753061 (601h15m56.099836117s) [noise=true , address=64.49.13.26:4947 , piece_id=BHVQNSBU3LP35Z5WQYTRPJZNNWTDZKWYK2CMRLHSOGBJ4IXGZ5SA, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.229701116 (601h15m56.100784151s) [noise=true , address=45.91.116.8:4071 , piece_id=3PUQCDJAXSQGEM47BTD2B5HWWNK3PUS6AEVQ63BEC2Q2CKCB4PXQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.230399837 (601h15m56.101482882s) [noise=true , address=77.90.31.21:4899 , piece_id=DCYPDKTC4S5LG3XEBQWZT3HGFJ63T3UC3PFUDLF7OOJHNMYW4KRQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.230834926 (601h15m56.101917971s) [noise=true , address=77.90.31.19:4853 , piece_id=MS4HNXUQIITLBMEHZDPJMEH7FSLMBBGKJSIP5BY2BDUF3IE2NKIQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.232511007 (601h15m56.103594112s) [noise=true , address=145.79.227.8:4344 , piece_id=6L5DD7CQJ5JZ7IPFX2IGHNEP7M6BZU2DA5Z5GEOKGPAMVZUXQHOQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.233047428 (601h15m56.104130473s) [noise=true , address=145.79.227.31:4808 , piece_id=MW4UFSJ4XFLXK7ZLWZFWTP2N4QQUS24WP3O44DJWT7AK2WCWAT3Q, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.233557500 (601h15m56.104640545s) [noise=true , address=145.79.227.33:4903 , piece_id=THSFTSO2Y4QVWIGNUOU4AKFGTSQWQ6FLNIIDISIVNRF7XZZWWPCQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.234065937 (601h15m56.105148983s) [noise=true , address=64.49.13.137:4493 , piece_id=QZSPIHMAGKSHSAJYJ6BR26EMTCWFAEPOPC4O65YN6SL6222H4HWQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.234513860 (601h15m56.105596896s) [noise=true , address=64.49.13.27:4965 , piece_id=AQTJD7BXKPMIAGEOGJCY5Y64IVU2SHCZEFJL2NLQAHVX5HR33N7Q, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.236125519 (601h15m56.107208554s) [noise=true , address=77.90.31.11:4567 , piece_id=NMCHY46TZ6W6EPVLVVCGQXHFERHBLSFB3J55AD32KQSKOVXABO7A, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.237179756 (601h15m56.108262791s) [noise=true , address=145.79.227.40:4064 , piece_id=SEYSDBZHX6SHMFZBGY3XDIYEOTWFCNGG2KNNTCIXYQRZP7FZOVJQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.238439144 (601h15m56.109522189s) [noise=true , address=145.79.227.30:4790 , piece_id=5VRI4FYK7J54TCOY7DX6TCQ3634N3TATGDZUZDWRBBLMXCGRVR3Q, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.239316585 (601h15m56.11039962s) [noise=true , address=145.79.227.44:4212 , piece_id=ZV3XMTVGJR2D5X4EEGGDTGGGXSQI4QNCJZCI3GPR6RTD7JEQWLGA, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.242138468 (601h15m56.113221503s) [noise=true , address=45.91.116.11:4131 , piece_id=YAFIDLBM762GWTNKDW62FMNLE4YHCNYW2WU3R4AMO5NELMVOHHHQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.243965547 (601h15m56.115048582s) [noise=true , address=77.90.31.18:4829 , piece_id=MVRLWV5URMPV57R52J426SXY5ZWQGHOTL42GO2GPEZ4LA7B2JGCQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.250805150 (601h15m56.121888185s) [noise=true , address=5.231.73.4:4033 , piece_id=CCH6AH47YBQPGPF4MPZU2NPT7HPF4D3KT3ZVDITRYF7UW5JUKWDQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.254973396 (601h15m56.126056431s) [noise=true , address=145.79.227.49:4341 , piece_id=6GKQQIBOECLDEEDZNO5AYKX7XMXZYUA6WNGKD3FQSF2Q4O6A4MQA, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:57.691193696 (601h15m57.562276731s) [upload=526987]: Next returned nil segment
Dec 30 16:44:57.691206460 (601h15m57.562289495s) [upload=526987]: Waiting for error group managing segments...
Dec 30 16:44:58.085111563 (601h15m57.956194598s) [noise=true , address=64.49.13.137:4493 , piece_id=QZSPIHMAGKSHSAJYJ6BR26EMTCWFAEPOPC4O65YN6SL6222H4HWQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.086614434 (601h15m57.957697479s) [noise=true , address=64.49.13.27:4965 , piece_id=AQTJD7BXKPMIAGEOGJCY5Y64IVU2SHCZEFJL2NLQAHVX5HR33N7Q, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.119421281 (601h15m57.990504316s) [noise=true , address=145.79.227.8:4344 , piece_id=6L5DD7CQJ5JZ7IPFX2IGHNEP7M6BZU2DA5Z5GEOKGPAMVZUXQHOQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.125317667 (601h15m57.996400702s) [noise=true , address=145.79.227.31:4808 , piece_id=MW4UFSJ4XFLXK7ZLWZFWTP2N4QQUS24WP3O44DJWT7AK2WCWAT3Q, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.262371457 (601h15m58.133454502s) [noise=true , address=64.49.13.26:4947 , piece_id=BHVQNSBU3LP35Z5WQYTRPJZNNWTDZKWYK2CMRLHSOGBJ4IXGZ5SA, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.284980216 (601h15m58.156063251s) [noise=true , address=77.90.31.19:4853 , piece_id=MS4HNXUQIITLBMEHZDPJMEH7FSLMBBGKJSIP5BY2BDUF3IE2NKIQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.387549123 (601h15m58.258632158s) [noise=true , address=77.90.31.18:4829 , piece_id=MVRLWV5URMPV57R52J426SXY5ZWQGHOTL42GO2GPEZ4LA7B2JGCQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.394280961 (601h15m58.265364006s) [noise=true , address=5.231.73.4:4033 , piece_id=CCH6AH47YBQPGPF4MPZU2NPT7HPF4D3KT3ZVDITRYF7UW5JUKWDQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.510328564 (601h15m58.381411609s) [noise=true , address=45.91.116.8:4071 , piece_id=3PUQCDJAXSQGEM47BTD2B5HWWNK3PUS6AEVQ63BEC2Q2CKCB4PXQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.545484694 (601h15m58.416567739s) [noise=true , address=45.91.116.11:4131 , piece_id=YAFIDLBM762GWTNKDW62FMNLE4YHCNYW2WU3R4AMO5NELMVOHHHQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.710409562 (601h15m58.581492607s) [noise=true , address=77.90.31.21:4899 , piece_id=DCYPDKTC4S5LG3XEBQWZT3HGFJ63T3UC3PFUDLF7OOJHNMYW4KRQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.981893937 (601h15m58.852976982s) [noise=true , address=145.79.227.44:4212 , piece_id=ZV3XMTVGJR2D5X4EEGGDTGGGXSQI4QNCJZCI3GPR6RTD7JEQWLGA, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.995663946 (601h15m58.866746991s) [noise=true , address=145.79.227.49:4341 , piece_id=6GKQQIBOECLDEEDZNO5AYKX7XMXZYUA6WNGKD3FQSF2Q4O6A4MQA, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:59.009469091 (601h15m58.880552136s) [noise=true , address=145.79.227.40:4064 , piece_id=SEYSDBZHX6SHMFZBGY3XDIYEOTWFCNGG2KNNTCIXYQRZP7FZOVJQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:59.009489950 (601h15m58.880572986s) [seg_pos={0 0}, upload=526987]: Segment reached optimal threshold of 14 pieces.
Dec 30 16:44:59.009662329 (601h15m58.880745374s) [noise=true , address=145.79.227.34:4923 , piece_id=OB3KQEHYXE4BL73EDDQDN2LSVYJYYQQFUSFJ655BFJISLBGOH4XQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VXAxRGiJQor24W7DYSeUQUu1QF7EcX6BQYgfUheACRs2quFFu): protocol: expected piece hash; context canceled; EOF
Dec 30 16:44:59.009668390 (601h15m58.880751425s) [noise=true , address=145.79.227.34:4923 , piece_id=OB3KQEHYXE4BL73EDDQDN2LSVYJYYQQFUSFJ655BFJISLBGOH4XQ, seg_pos={0 0}, upload=526987]: Long tail cancellation due to optimal threshold
Dec 30 16:44:59.009857671 (601h15m58.880940706s) [noise=true , address=145.79.227.33:4903 , piece_id=THSFTSO2Y4QVWIGNUOU4AKFGTSQWQ6FLNIIDISIVNRF7XZZWWPCQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:59.009921232 (601h15m58.881004267s) [noise=true , address=145.79.227.30:4790 , piece_id=5VRI4FYK7J54TCOY7DX6TCQ3634N3TATGDZUZDWRBBLMXCGRVR3Q, seg_pos={0 0}, upload=526987]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kuSMof8EbvsaMCdVxKWjgTT5TfLJnTAeHUf8DKNC3UGQn5q7T): protocol: expected piece hash; context canceled; EOF
Dec 30 16:44:59.009927314 (601h15m58.881010359s) [noise=true , address=145.79.227.30:4790 , piece_id=5VRI4FYK7J54TCOY7DX6TCQ3634N3TATGDZUZDWRBBLMXCGRVR3Q, seg_pos={0 0}, upload=526987]: Long tail cancellation due to optimal threshold
Dec 30 16:44:59.010092589 (601h15m58.881175624s) [noise=true , address=77.90.31.11:4567 , piece_id=NMCHY46TZ6W6EPVLVVCGQXHFERHBLSFB3J55AD32KQSKOVXABO7A, seg_pos={0 0}, upload=526987]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HJVmEuxjrwWCcdwxWYFC5fTULds1kiRH82DiRWipWhRT1Qwdk): protocol: expected piece hash; context canceled; EOF
Dec 30 16:44:59.010101946 (601h15m58.881184981s) [noise=true , address=77.90.31.11:4567 , piece_id=NMCHY46TZ6W6EPVLVVCGQXHFERHBLSFB3J55AD32KQSKOVXABO7A, seg_pos={0 0}, upload=526987]: Long tail cancellation due to optimal threshold
Dec 30 16:44:59.010125410 (601h15m58.881208445s) [seg_pos={0 0}, upload=526987]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil>
Dec 30 16:44:59.010204091 (601h15m58.881287136s) [upload=526987]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 16:44:59.010211274 (601h15m58.881294339s) [upload=526987]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 16:44:59.309660956 (601h15m59.180744001s) [upload=526987]: Done uploading segments...
Dec 30 16:44:59.309679662 (601h15m59.180762697s) [upload=526987]: Upload finished. err: <nil>
Duration: 23.855657912s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [............X.............X.......X.........................] duration:
15.260744192s trace:
https://jaeger.dev.storj.io/trace/3f4379d3637c8e4b
FAIL (2025-12-30 17:04:55 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1WBuBdGxdPXi6VsNf4FJpn3LpWoQztM7CV6YjAW1mUDVjAUjn2, Piece ID: MS4HNXUQIITLBMEHZDPJMEH7FSLMBBGKJSIP5BY2BDUF3IE2NKIQ) context deadline exceeded; piecestore: (Node ID: 1soiTQsiMNe5BkNHwwAGykmtP9LtLxncqE7Tv728ak8pjFeshg, Piece ID: BHVQNSBU3LP35Z5WQYTRPJZNNWTDZKWYK2CMRLHSOGBJ4IXGZ5SA) context deadline exceeded; piecestore: (Node ID: 1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, Piece ID: MVRLWV5URMPV57R52J426SXY5ZWQGHOTL42GO2GPEZ4LA7B2JGCQ) context deadline exceeded
======LOGS======
5m56.099625936s) [seg_pos={0 0}, upload=526987]: Begin upload segment...
Dec 30 16:44:56.228605921 (601h15m56.099688967s) [seg_pos={0 0}, upload=526987]: Done begin upload segment.
Dec 30 16:44:56.228736641 (601h15m56.099819686s) [noise=true , address=145.79.227.34:4923 , piece_id=OB3KQEHYXE4BL73EDDQDN2LSVYJYYQQFUSFJ655BFJISLBGOH4XQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.228753061 (601h15m56.099836117s) [noise=true , address=64.49.13.26:4947 , piece_id=BHVQNSBU3LP35Z5WQYTRPJZNNWTDZKWYK2CMRLHSOGBJ4IXGZ5SA, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.229701116 (601h15m56.100784151s) [noise=true , address=45.91.116.8:4071 , piece_id=3PUQCDJAXSQGEM47BTD2B5HWWNK3PUS6AEVQ63BEC2Q2CKCB4PXQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.230399837 (601h15m56.101482882s) [noise=true , address=77.90.31.21:4899 , piece_id=DCYPDKTC4S5LG3XEBQWZT3HGFJ63T3UC3PFUDLF7OOJHNMYW4KRQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.230834926 (601h15m56.101917971s) [noise=true , address=77.90.31.19:4853 , piece_id=MS4HNXUQIITLBMEHZDPJMEH7FSLMBBGKJSIP5BY2BDUF3IE2NKIQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.232511007 (601h15m56.103594112s) [noise=true , address=145.79.227.8:4344 , piece_id=6L5DD7CQJ5JZ7IPFX2IGHNEP7M6BZU2DA5Z5GEOKGPAMVZUXQHOQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.233047428 (601h15m56.104130473s) [noise=true , address=145.79.227.31:4808 , piece_id=MW4UFSJ4XFLXK7ZLWZFWTP2N4QQUS24WP3O44DJWT7AK2WCWAT3Q, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.233557500 (601h15m56.104640545s) [noise=true , address=145.79.227.33:4903 , piece_id=THSFTSO2Y4QVWIGNUOU4AKFGTSQWQ6FLNIIDISIVNRF7XZZWWPCQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.234065937 (601h15m56.105148983s) [noise=true , address=64.49.13.137:4493 , piece_id=QZSPIHMAGKSHSAJYJ6BR26EMTCWFAEPOPC4O65YN6SL6222H4HWQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.234513860 (601h15m56.105596896s) [noise=true , address=64.49.13.27:4965 , piece_id=AQTJD7BXKPMIAGEOGJCY5Y64IVU2SHCZEFJL2NLQAHVX5HR33N7Q, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.236125519 (601h15m56.107208554s) [noise=true , address=77.90.31.11:4567 , piece_id=NMCHY46TZ6W6EPVLVVCGQXHFERHBLSFB3J55AD32KQSKOVXABO7A, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.237179756 (601h15m56.108262791s) [noise=true , address=145.79.227.40:4064 , piece_id=SEYSDBZHX6SHMFZBGY3XDIYEOTWFCNGG2KNNTCIXYQRZP7FZOVJQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.238439144 (601h15m56.109522189s) [noise=true , address=145.79.227.30:4790 , piece_id=5VRI4FYK7J54TCOY7DX6TCQ3634N3TATGDZUZDWRBBLMXCGRVR3Q, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.239316585 (601h15m56.11039962s) [noise=true , address=145.79.227.44:4212 , piece_id=ZV3XMTVGJR2D5X4EEGGDTGGGXSQI4QNCJZCI3GPR6RTD7JEQWLGA, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.242138468 (601h15m56.113221503s) [noise=true , address=45.91.116.11:4131 , piece_id=YAFIDLBM762GWTNKDW62FMNLE4YHCNYW2WU3R4AMO5NELMVOHHHQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.243965547 (601h15m56.115048582s) [noise=true , address=77.90.31.18:4829 , piece_id=MVRLWV5URMPV57R52J426SXY5ZWQGHOTL42GO2GPEZ4LA7B2JGCQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.250805150 (601h15m56.121888185s) [noise=true , address=5.231.73.4:4033 , piece_id=CCH6AH47YBQPGPF4MPZU2NPT7HPF4D3KT3ZVDITRYF7UW5JUKWDQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.254973396 (601h15m56.126056431s) [noise=true , address=145.79.227.49:4341 , piece_id=6GKQQIBOECLDEEDZNO5AYKX7XMXZYUA6WNGKD3FQSF2Q4O6A4MQA, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:57.691193696 (601h15m57.562276731s) [upload=526987]: Next returned nil segment
Dec 30 16:44:57.691206460 (601h15m57.562289495s) [upload=526987]: Waiting for error group managing segments...
Dec 30 16:44:58.085111563 (601h15m57.956194598s) [noise=true , address=64.49.13.137:4493 , piece_id=QZSPIHMAGKSHSAJYJ6BR26EMTCWFAEPOPC4O65YN6SL6222H4HWQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.086614434 (601h15m57.957697479s) [noise=true , address=64.49.13.27:4965 , piece_id=AQTJD7BXKPMIAGEOGJCY5Y64IVU2SHCZEFJL2NLQAHVX5HR33N7Q, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.119421281 (601h15m57.990504316s) [noise=true , address=145.79.227.8:4344 , piece_id=6L5DD7CQJ5JZ7IPFX2IGHNEP7M6BZU2DA5Z5GEOKGPAMVZUXQHOQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.125317667 (601h15m57.996400702s) [noise=true , address=145.79.227.31:4808 , piece_id=MW4UFSJ4XFLXK7ZLWZFWTP2N4QQUS24WP3O44DJWT7AK2WCWAT3Q, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.262371457 (601h15m58.133454502s) [noise=true , address=64.49.13.26:4947 , piece_id=BHVQNSBU3LP35Z5WQYTRPJZNNWTDZKWYK2CMRLHSOGBJ4IXGZ5SA, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.284980216 (601h15m58.156063251s) [noise=true , address=77.90.31.19:4853 , piece_id=MS4HNXUQIITLBMEHZDPJMEH7FSLMBBGKJSIP5BY2BDUF3IE2NKIQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.387549123 (601h15m58.258632158s) [noise=true , address=77.90.31.18:4829 , piece_id=MVRLWV5URMPV57R52J426SXY5ZWQGHOTL42GO2GPEZ4LA7B2JGCQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.394280961 (601h15m58.265364006s) [noise=true , address=5.231.73.4:4033 , piece_id=CCH6AH47YBQPGPF4MPZU2NPT7HPF4D3KT3ZVDITRYF7UW5JUKWDQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.510328564 (601h15m58.381411609s) [noise=true , address=45.91.116.8:4071 , piece_id=3PUQCDJAXSQGEM47BTD2B5HWWNK3PUS6AEVQ63BEC2Q2CKCB4PXQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.545484694 (601h15m58.416567739s) [noise=true , address=45.91.116.11:4131 , piece_id=YAFIDLBM762GWTNKDW62FMNLE4YHCNYW2WU3R4AMO5NELMVOHHHQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.710409562 (601h15m58.581492607s) [noise=true , address=77.90.31.21:4899 , piece_id=DCYPDKTC4S5LG3XEBQWZT3HGFJ63T3UC3PFUDLF7OOJHNMYW4KRQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.981893937 (601h15m58.852976982s) [noise=true , address=145.79.227.44:4212 , piece_id=ZV3XMTVGJR2D5X4EEGGDTGGGXSQI4QNCJZCI3GPR6RTD7JEQWLGA, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.995663946 (601h15m58.866746991s) [noise=true , address=145.79.227.49:4341 , piece_id=6GKQQIBOECLDEEDZNO5AYKX7XMXZYUA6WNGKD3FQSF2Q4O6A4MQA, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:59.009469091 (601h15m58.880552136s) [noise=true , address=145.79.227.40:4064 , piece_id=SEYSDBZHX6SHMFZBGY3XDIYEOTWFCNGG2KNNTCIXYQRZP7FZOVJQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:59.009489950 (601h15m58.880572986s) [seg_pos={0 0}, upload=526987]: Segment reached optimal threshold of 14 pieces.
Dec 30 16:44:59.009662329 (601h15m58.880745374s) [noise=true , address=145.79.227.34:4923 , piece_id=OB3KQEHYXE4BL73EDDQDN2LSVYJYYQQFUSFJ655BFJISLBGOH4XQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VXAxRGiJQor24W7DYSeUQUu1QF7EcX6BQYgfUheACRs2quFFu): protocol: expected piece hash; context canceled; EOF
Dec 30 16:44:59.009668390 (601h15m58.880751425s) [noise=true , address=145.79.227.34:4923 , piece_id=OB3KQEHYXE4BL73EDDQDN2LSVYJYYQQFUSFJ655BFJISLBGOH4XQ, seg_pos={0 0}, upload=526987]: Long tail cancellation due to optimal threshold
Dec 30 16:44:59.009857671 (601h15m58.880940706s) [noise=true , address=145.79.227.33:4903 , piece_id=THSFTSO2Y4QVWIGNUOU4AKFGTSQWQ6FLNIIDISIVNRF7XZZWWPCQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:59.009921232 (601h15m58.881004267s) [noise=true , address=145.79.227.30:4790 , piece_id=5VRI4FYK7J54TCOY7DX6TCQ3634N3TATGDZUZDWRBBLMXCGRVR3Q, seg_pos={0 0}, upload=526987]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kuSMof8EbvsaMCdVxKWjgTT5TfLJnTAeHUf8DKNC3UGQn5q7T): protocol: expected piece hash; context canceled; EOF
Dec 30 16:44:59.009927314 (601h15m58.881010359s) [noise=true , address=145.79.227.30:4790 , piece_id=5VRI4FYK7J54TCOY7DX6TCQ3634N3TATGDZUZDWRBBLMXCGRVR3Q, seg_pos={0 0}, upload=526987]: Long tail cancellation due to optimal threshold
Dec 30 16:44:59.010092589 (601h15m58.881175624s) [noise=true , address=77.90.31.11:4567 , piece_id=NMCHY46TZ6W6EPVLVVCGQXHFERHBLSFB3J55AD32KQSKOVXABO7A, seg_pos={0 0}, upload=526987]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HJVmEuxjrwWCcdwxWYFC5fTULds1kiRH82DiRWipWhRT1Qwdk): protocol: expected piece hash; context canceled; EOF
Dec 30 16:44:59.010101946 (601h15m58.881184981s) [noise=true , address=77.90.31.11:4567 , piece_id=NMCHY46TZ6W6EPVLVVCGQXHFERHBLSFB3J55AD32KQSKOVXABO7A, seg_pos={0 0}, upload=526987]: Long tail cancellation due to optimal threshold
Dec 30 16:44:59.010125410 (601h15m58.881208445s) [seg_pos={0 0}, upload=526987]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil>
Dec 30 16:44:59.010204091 (601h15m58.881287136s) [upload=526987]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 16:44:59.010211274 (601h15m58.881294339s) [upload=526987]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 16:44:59.309660956 (601h15m59.180744001s) [upload=526987]: Done uploading segments...
Dec 30 16:44:59.309679662 (601h15m59.180762697s) [upload=526987]: Upload finished. err: <nil>
Duration: 15.260773979s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
19 out of 60 is failed (threshold=6) [XX.X.X.X...XXX..X....XXX..X...XX......X.XX................X.] duration:
3.602085139s trace:
https://jaeger.dev.storj.io/trace/16a0763f8dcf089e
FAIL (2025-12-30 17:04:33 UTC)
orbiter warning: context deadline exceeded
Duration: 3.602115176s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
24 out of 60 is failed (threshold=3) [XXX........X..X....X.XXX..XX.X.X..XXX.XX..X.X.......X.X..XX.] duration:
6.287016053s trace:
https://jaeger.dev.storj.io/trace/411e3d14de7b3865
FAIL (2025-12-30 17:04:03 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 30 17:03:18.654418548 (601h34m18.525501583s) [upload=527254]: Starting upload
Dec 30 17:03:18.654472180 (601h34m18.525555215s) [upload=527254]: Done starting upload
Dec 30 17:03:18.654572972 (601h34m18.525656017s) [upload=527254]: Uploading segments...
Dec 30 17:03:18.654725121 (601h34m18.525808156s) [upload=527254]: Got next segment. Inline: false
Dec 30 17:03:18.654775547 (601h34m18.525858583s) [upload=527254]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 17:03:18.654781308 (601h34m18.525864353s) [upload=527254]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 17:03:18.901136047 (601h34m18.772219102s) [seg_pos={0 0}, upload=527254]: Begin upload segment...
Dec 30 17:03:18.901196763 (601h34m18.772279808s) [seg_pos={0 0}, upload=527254]: Done begin upload segment.
Dec 30 17:03:18.901226700 (601h34m18.772309735s) [noise=true , address=45.91.116.10:4081 , piece_id=4Z5V7GUGUH3EIQ7TQTUMW7T3Z4GYZHKABTP3G5Z2KXLX4NXHZO2A, seg_pos={0 0}, upload=527254]: Uploading piece...
Dec 30 17:03:18.901309127 (601h34m18.772392172s) [noise=true , address=145.79.227.17:4970 , piece_id=SHRTIGRS6KK6PW3TD2JOVMFHD7ZKDQ2IMOGZB4QYBZTMMJL6XMEQ, seg_pos={0 0}, upload=527254]: Uploading piece...
Dec 30 17:03:18.901758002 (601h34m18.772841037s) [noise=true , address=145.79.227.3:4149 , piece_id=LNIGGIEGL2F3SCYTCCOEZKBSBEV3GSYFGWEHPGP7UWBANW5VPQCQ, seg_pos={0 0}, upload=527254]: Uploading piece...
Dec 30 17:03:18.902077350 (601h34m18.773160385s) [noise=true , address=145.79.227.50:4373 , piece_id=N6GITFLLTPU7A32DIPBGVAAPAH6AJE6E2NTMR4S4SXGIUECLAN2Q, seg_pos={0 0}, upload=527254]: Uploading piece...
Dec 30 17:03:18.902360430 (601h34m18.773443465s) [noise=true , address=145.79.227.15:4876 , piece_id=V2OXOBNCJYYLVA3WW55BXZ5VK6LS2NVKYYLPL46O7TH53H4VEBMA, seg_pos={0 0}, upload=527254]: Uploading piece...
Dec 30 17:03:18.902708793 (601h34m18.773791828s) [noise=true , address=77.90.31.18:4805 , piece_id=7DCRBKUNZJ2DSDS26WRNRV6JTHBZA23A4L6S5VSRU7PMASBKHSXA, seg_pos={0 0}, upload=527254]: Uploading piece...
Dec 30 17:03:18.903009074 (601h34m18.774092109s) [noise=true , address=145.79.227.45:4230 , piece_id=QTBZMM7G37DLTHPVBROH37RRXQYHT7PQB4AQQ7H6TXSTTE5XFMNQ, seg_pos={0 0}, upload=527254]: Uploading piece...
Dec 30 17:03:18.903293646 (601h34m18.774376681s) [noise=true , address=145.79.227.23:4545 , piece_id=4O6USJ7XQ3SSYCIV5BZ6AIYQLSFAIROYLEHGUDDVZQDOSREL4NUA, seg_pos={0 0}, upload=527254]: Uploading piece...
Dec 30 17:03:18.903640626 (601h34m18.774723662s) [noise=true , address=145.79.227.48:4302 , piece_id=JLPS26KBBMNG6ZSNMOYZPUF5OLYE64JUZRTQQBD7BQCC4ABJTMSQ, seg_pos={0 0}, upload=527254]: Uploading piece...
Dec 30 17:03:18.905010977 (601h34m18.776094012s) [noise=true , address=145.79.227.10:4402 , piece_id=D2EOI6P3G3REU4E5EIOINVUIXWCLIL7W7UOORJBPXVE6347EIZQA, seg_pos={0 0}, upload=527254]: Uploading piece...
Dec 30 17:03:18.906509580 (601h34m18.777592615s) [noise=true , address=145.79.227.24:4581 , piece_id=DU2BRTNM2GJVG2RY2BI3Y53342UW3SE7BC6UDX7Y4HRSHCYXUVCA, seg_pos={0 0}, upload=527254]: Uploading piece...
Dec 30 17:03:18.907799827 (601h34m18.778882862s) [noise=true , address=77.90.31.12:4589 , piece_id=ZPQWP76WNPDCG5LV3TFZJDWSJSMJKPJSKBRBZQQ3NGY2JK7WSANA, seg_pos={0 0}, upload=527254]: Uploading piece...
Dec 30 17:03:18.909107506 (601h34m18.780190541s) [noise=true , address=45.91.116.4:4937 , piece_id=2IN6YCNOFSZE4SO5A7FH4662UFBTSKKAKWP7VUIEZQXNME4GQY3A, seg_pos={0 0}, upload=527254]: Uploading piece...
Dec 30 17:03:18.910473167 (601h34m18.781556202s) [noise=true , address=145.79.227.25:4617 , piece_id=4QAQOAFIYOMPZPMBUKMSNHZDLMSSUP6XTAHUEIJOG7IRXOD2764A, seg_pos={0 0}, upload=527254]: Uploading piece...
Dec 30 17:03:18.911857843 (601h34m18.782940878s) [noise=true , address=145.79.227.4:4185 , piece_id=RGJDDGO7HLKKIKG2KM5ITAWTXSOWCUO4PC6QTZNQ63MNYIAFTURA, seg_pos={0 0}, upload=527254]: Uploading piece...
Dec 30 17:03:18.912354930 (601h34m18.783437975s) [noise=true , address=77.90.31.10:4517 , piece_id=XGYFOBF3RWEDX4HDLDJU4SSYWQ3RLXGSQVJNKF5VCU777CADCVCA, seg_pos={0 0}, upload=527254]: Uploading piece...
Dec 30 17:03:20.603031825 (601h34m20.47411487s) [upload=527254]: Next returned nil segment
Dec 30 17:03:20.603079595 (601h34m20.474162631s) [upload=527254]: Waiting for error group managing segments...
Dec 30 17:03:20.882104073 (601h34m20.753187108s) [noise=true , address=145.79.227.50:4373 , piece_id=N6GITFLLTPU7A32DIPBGVAAPAH6AJE6E2NTMR4S4SXGIUECLAN2Q, seg_pos={0 0}, upload=527254]: Done uploading piece. err: <nil>
Dec 30 17:03:20.888923718 (601h34m20.760006763s) [noise=true , address=145.79.227.23:4545 , piece_id=4O6USJ7XQ3SSYCIV5BZ6AIYQLSFAIROYLEHGUDDVZQDOSREL4NUA, seg_pos={0 0}, upload=527254]: Done uploading piece. err: <nil>
Dec 30 17:03:20.894580499 (601h34m20.765663544s) [noise=true , address=145.79.227.3:4149 , piece_id=LNIGGIEGL2F3SCYTCCOEZKBSBEV3GSYFGWEHPGP7UWBANW5VPQCQ, seg_pos={0 0}, upload=527254]: Done uploading piece. err: <nil>
Dec 30 17:03:20.896608281 (601h34m20.767691326s) [noise=true , address=145.79.227.15:4876 , piece_id=V2OXOBNCJYYLVA3WW55BXZ5VK6LS2NVKYYLPL46O7TH53H4VEBMA, seg_pos={0 0}, upload=527254]: Done uploading piece. err: <nil>
Dec 30 17:03:20.899132577 (601h34m20.770215622s) [noise=true , address=145.79.227.17:4970 , piece_id=SHRTIGRS6KK6PW3TD2JOVMFHD7ZKDQ2IMOGZB4QYBZTMMJL6XMEQ, seg_pos={0 0}, upload=527254]: Done uploading piece. err: <nil>
Dec 30 17:03:20.901263484 (601h34m20.772346529s) [noise=true , address=145.79.227.24:4581 , piece_id=DU2BRTNM2GJVG2RY2BI3Y53342UW3SE7BC6UDX7Y4HRSHCYXUVCA, seg_pos={0 0}, upload=527254]: Done uploading piece. err: <nil>
Dec 30 17:03:20.908995056 (601h34m20.780078091s) [noise=true , address=145.79.227.10:4402 , piece_id=D2EOI6P3G3REU4E5EIOINVUIXWCLIL7W7UOORJBPXVE6347EIZQA, seg_pos={0 0}, upload=527254]: Done uploading piece. err: <nil>
Dec 30 17:03:20.911370389 (601h34m20.782453434s) [noise=true , address=145.79.227.4:4185 , piece_id=RGJDDGO7HLKKIKG2KM5ITAWTXSOWCUO4PC6QTZNQ63MNYIAFTURA, seg_pos={0 0}, upload=527254]: Done uploading piece. err: <nil>
Dec 30 17:03:20.917279470 (601h34m20.788362505s) [noise=true , address=145.79.227.48:4302 , piece_id=JLPS26KBBMNG6ZSNMOYZPUF5OLYE64JUZRTQQBD7BQCC4ABJTMSQ, seg_pos={0 0}, upload=527254]: Done uploading piece. err: <nil>
Dec 30 17:03:20.920107986 (601h34m20.791191031s) [noise=true , address=145.79.227.45:4230 , piece_id=QTBZMM7G37DLTHPVBROH37RRXQYHT7PQB4AQQ7H6TXSTTE5XFMNQ, seg_pos={0 0}, upload=527254]: Done uploading piece. err: <nil>
Dec 30 17:03:20.944777200 (601h34m20.815860235s) [noise=true , address=145.79.227.25:4617 , piece_id=4QAQOAFIYOMPZPMBUKMSNHZDLMSSUP6XTAHUEIJOG7IRXOD2764A, seg_pos={0 0}, upload=527254]: Done uploading piece. err: <nil>
Dec 30 17:03:21.095378983 (601h34m20.966462028s) [noise=true , address=77.90.31.12:4589 , piece_id=ZPQWP76WNPDCG5LV3TFZJDWSJSMJKPJSKBRBZQQ3NGY2JK7WSANA, seg_pos={0 0}, upload=527254]: Done uploading piece. err: <nil>
Dec 30 17:03:21.476737317 (601h34m21.347820352s) [noise=true , address=77.90.31.10:4517 , piece_id=XGYFOBF3RWEDX4HDLDJU4SSYWQ3RLXGSQVJNKF5VCU777CADCVCA, seg_pos={0 0}, upload=527254]: Done uploading piece. err: <nil>
Dec 30 17:03:21.476766633 (601h34m21.347849668s) [seg_pos={0 0}, upload=527254]: Segment reached optimal threshold of 13 pieces.
Dec 30 17:03:21.476995379 (601h34m21.348078424s) [noise=true , address=45.91.116.10:4081 , piece_id=4Z5V7GUGUH3EIQ7TQTUMW7T3Z4GYZHKABTP3G5Z2KXLX4NXHZO2A, seg_pos={0 0}, upload=527254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:35034->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:35034->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35034->45.91.116.10:4081: use of closed network connection
Dec 30 17:03:21.477004907 (601h34m21.348087942s) [noise=true , address=45.91.116.10:4081 , piece_id=4Z5V7GUGUH3EIQ7TQTUMW7T3Z4GYZHKABTP3G5Z2KXLX4NXHZO2A, seg_pos={0 0}, upload=527254]: Long tail cancellation due to optimal threshold
Dec 30 17:03:21.477175773 (601h34m21.348258808s) [noise=true , address=77.90.31.18:4805 , piece_id=7DCRBKUNZJ2DSDS26WRNRV6JTHBZA23A4L6S5VSRU7PMASBKHSXA, seg_pos={0 0}, upload=527254]: Done uploading piece. err: <nil>
Dec 30 17:03:21.477360594 (601h34m21.348443629s) [noise=true , address=45.91.116.4:4937 , piece_id=2IN6YCNOFSZE4SO5A7FH4662UFBTSKKAKWP7VUIEZQXNME4GQY3A, seg_pos={0 0}, upload=527254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF
Dec 30 17:03:21.477371144 (601h34m21.348454189s) [noise=true , address=45.91.116.4:4937 , piece_id=2IN6YCNOFSZE4SO5A7FH4662UFBTSKKAKWP7VUIEZQXNME4GQY3A, seg_pos={0 0}, upload=527254]: Long tail cancellation due to optimal threshold
Dec 30 17:03:21.477392435 (601h34m21.34847547s) [seg_pos={0 0}, upload=527254]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil>
Dec 30 17:03:21.477497255 (601h34m21.3485803s) [upload=527254]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 17:03:21.477506582 (601h34m21.348589637s) [upload=527254]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 17:03:21.761466016 (601h34m21.632549071s) [upload=527254]: Done uploading segments...
Dec 30 17:03:21.761492797 (601h34m21.632575842s) [upload=527254]: Upload finished. err: <nil>
Duration: 6.287181228s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [.............X.............X.......X........................] duration:
34.341319454s trace:
https://jaeger.dev.storj.io/trace/3f4379d3637c8e4b
FAIL (2025-12-30 17:04:01 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1WBuBdGxdPXi6VsNf4FJpn3LpWoQztM7CV6YjAW1mUDVjAUjn2, Piece ID: MS4HNXUQIITLBMEHZDPJMEH7FSLMBBGKJSIP5BY2BDUF3IE2NKIQ) context deadline exceeded; piecestore: (Node ID: 1soiTQsiMNe5BkNHwwAGykmtP9LtLxncqE7Tv728ak8pjFeshg, Piece ID: BHVQNSBU3LP35Z5WQYTRPJZNNWTDZKWYK2CMRLHSOGBJ4IXGZ5SA) context deadline exceeded; piecestore: (Node ID: 1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, Piece ID: MVRLWV5URMPV57R52J426SXY5ZWQGHOTL42GO2GPEZ4LA7B2JGCQ) context deadline exceeded
======LOGS======
5m56.099625936s) [seg_pos={0 0}, upload=526987]: Begin upload segment...
Dec 30 16:44:56.228605921 (601h15m56.099688967s) [seg_pos={0 0}, upload=526987]: Done begin upload segment.
Dec 30 16:44:56.228736641 (601h15m56.099819686s) [noise=true , address=145.79.227.34:4923 , piece_id=OB3KQEHYXE4BL73EDDQDN2LSVYJYYQQFUSFJ655BFJISLBGOH4XQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.228753061 (601h15m56.099836117s) [noise=true , address=64.49.13.26:4947 , piece_id=BHVQNSBU3LP35Z5WQYTRPJZNNWTDZKWYK2CMRLHSOGBJ4IXGZ5SA, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.229701116 (601h15m56.100784151s) [noise=true , address=45.91.116.8:4071 , piece_id=3PUQCDJAXSQGEM47BTD2B5HWWNK3PUS6AEVQ63BEC2Q2CKCB4PXQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.230399837 (601h15m56.101482882s) [noise=true , address=77.90.31.21:4899 , piece_id=DCYPDKTC4S5LG3XEBQWZT3HGFJ63T3UC3PFUDLF7OOJHNMYW4KRQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.230834926 (601h15m56.101917971s) [noise=true , address=77.90.31.19:4853 , piece_id=MS4HNXUQIITLBMEHZDPJMEH7FSLMBBGKJSIP5BY2BDUF3IE2NKIQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.232511007 (601h15m56.103594112s) [noise=true , address=145.79.227.8:4344 , piece_id=6L5DD7CQJ5JZ7IPFX2IGHNEP7M6BZU2DA5Z5GEOKGPAMVZUXQHOQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.233047428 (601h15m56.104130473s) [noise=true , address=145.79.227.31:4808 , piece_id=MW4UFSJ4XFLXK7ZLWZFWTP2N4QQUS24WP3O44DJWT7AK2WCWAT3Q, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.233557500 (601h15m56.104640545s) [noise=true , address=145.79.227.33:4903 , piece_id=THSFTSO2Y4QVWIGNUOU4AKFGTSQWQ6FLNIIDISIVNRF7XZZWWPCQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.234065937 (601h15m56.105148983s) [noise=true , address=64.49.13.137:4493 , piece_id=QZSPIHMAGKSHSAJYJ6BR26EMTCWFAEPOPC4O65YN6SL6222H4HWQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.234513860 (601h15m56.105596896s) [noise=true , address=64.49.13.27:4965 , piece_id=AQTJD7BXKPMIAGEOGJCY5Y64IVU2SHCZEFJL2NLQAHVX5HR33N7Q, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.236125519 (601h15m56.107208554s) [noise=true , address=77.90.31.11:4567 , piece_id=NMCHY46TZ6W6EPVLVVCGQXHFERHBLSFB3J55AD32KQSKOVXABO7A, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.237179756 (601h15m56.108262791s) [noise=true , address=145.79.227.40:4064 , piece_id=SEYSDBZHX6SHMFZBGY3XDIYEOTWFCNGG2KNNTCIXYQRZP7FZOVJQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.238439144 (601h15m56.109522189s) [noise=true , address=145.79.227.30:4790 , piece_id=5VRI4FYK7J54TCOY7DX6TCQ3634N3TATGDZUZDWRBBLMXCGRVR3Q, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.239316585 (601h15m56.11039962s) [noise=true , address=145.79.227.44:4212 , piece_id=ZV3XMTVGJR2D5X4EEGGDTGGGXSQI4QNCJZCI3GPR6RTD7JEQWLGA, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.242138468 (601h15m56.113221503s) [noise=true , address=45.91.116.11:4131 , piece_id=YAFIDLBM762GWTNKDW62FMNLE4YHCNYW2WU3R4AMO5NELMVOHHHQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.243965547 (601h15m56.115048582s) [noise=true , address=77.90.31.18:4829 , piece_id=MVRLWV5URMPV57R52J426SXY5ZWQGHOTL42GO2GPEZ4LA7B2JGCQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.250805150 (601h15m56.121888185s) [noise=true , address=5.231.73.4:4033 , piece_id=CCH6AH47YBQPGPF4MPZU2NPT7HPF4D3KT3ZVDITRYF7UW5JUKWDQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.254973396 (601h15m56.126056431s) [noise=true , address=145.79.227.49:4341 , piece_id=6GKQQIBOECLDEEDZNO5AYKX7XMXZYUA6WNGKD3FQSF2Q4O6A4MQA, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:57.691193696 (601h15m57.562276731s) [upload=526987]: Next returned nil segment
Dec 30 16:44:57.691206460 (601h15m57.562289495s) [upload=526987]: Waiting for error group managing segments...
Dec 30 16:44:58.085111563 (601h15m57.956194598s) [noise=true , address=64.49.13.137:4493 , piece_id=QZSPIHMAGKSHSAJYJ6BR26EMTCWFAEPOPC4O65YN6SL6222H4HWQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.086614434 (601h15m57.957697479s) [noise=true , address=64.49.13.27:4965 , piece_id=AQTJD7BXKPMIAGEOGJCY5Y64IVU2SHCZEFJL2NLQAHVX5HR33N7Q, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.119421281 (601h15m57.990504316s) [noise=true , address=145.79.227.8:4344 , piece_id=6L5DD7CQJ5JZ7IPFX2IGHNEP7M6BZU2DA5Z5GEOKGPAMVZUXQHOQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.125317667 (601h15m57.996400702s) [noise=true , address=145.79.227.31:4808 , piece_id=MW4UFSJ4XFLXK7ZLWZFWTP2N4QQUS24WP3O44DJWT7AK2WCWAT3Q, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.262371457 (601h15m58.133454502s) [noise=true , address=64.49.13.26:4947 , piece_id=BHVQNSBU3LP35Z5WQYTRPJZNNWTDZKWYK2CMRLHSOGBJ4IXGZ5SA, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.284980216 (601h15m58.156063251s) [noise=true , address=77.90.31.19:4853 , piece_id=MS4HNXUQIITLBMEHZDPJMEH7FSLMBBGKJSIP5BY2BDUF3IE2NKIQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.387549123 (601h15m58.258632158s) [noise=true , address=77.90.31.18:4829 , piece_id=MVRLWV5URMPV57R52J426SXY5ZWQGHOTL42GO2GPEZ4LA7B2JGCQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.394280961 (601h15m58.265364006s) [noise=true , address=5.231.73.4:4033 , piece_id=CCH6AH47YBQPGPF4MPZU2NPT7HPF4D3KT3ZVDITRYF7UW5JUKWDQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.510328564 (601h15m58.381411609s) [noise=true , address=45.91.116.8:4071 , piece_id=3PUQCDJAXSQGEM47BTD2B5HWWNK3PUS6AEVQ63BEC2Q2CKCB4PXQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.545484694 (601h15m58.416567739s) [noise=true , address=45.91.116.11:4131 , piece_id=YAFIDLBM762GWTNKDW62FMNLE4YHCNYW2WU3R4AMO5NELMVOHHHQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.710409562 (601h15m58.581492607s) [noise=true , address=77.90.31.21:4899 , piece_id=DCYPDKTC4S5LG3XEBQWZT3HGFJ63T3UC3PFUDLF7OOJHNMYW4KRQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.981893937 (601h15m58.852976982s) [noise=true , address=145.79.227.44:4212 , piece_id=ZV3XMTVGJR2D5X4EEGGDTGGGXSQI4QNCJZCI3GPR6RTD7JEQWLGA, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.995663946 (601h15m58.866746991s) [noise=true , address=145.79.227.49:4341 , piece_id=6GKQQIBOECLDEEDZNO5AYKX7XMXZYUA6WNGKD3FQSF2Q4O6A4MQA, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:59.009469091 (601h15m58.880552136s) [noise=true , address=145.79.227.40:4064 , piece_id=SEYSDBZHX6SHMFZBGY3XDIYEOTWFCNGG2KNNTCIXYQRZP7FZOVJQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:59.009489950 (601h15m58.880572986s) [seg_pos={0 0}, upload=526987]: Segment reached optimal threshold of 14 pieces.
Dec 30 16:44:59.009662329 (601h15m58.880745374s) [noise=true , address=145.79.227.34:4923 , piece_id=OB3KQEHYXE4BL73EDDQDN2LSVYJYYQQFUSFJ655BFJISLBGOH4XQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VXAxRGiJQor24W7DYSeUQUu1QF7EcX6BQYgfUheACRs2quFFu): protocol: expected piece hash; context canceled; EOF
Dec 30 16:44:59.009668390 (601h15m58.880751425s) [noise=true , address=145.79.227.34:4923 , piece_id=OB3KQEHYXE4BL73EDDQDN2LSVYJYYQQFUSFJ655BFJISLBGOH4XQ, seg_pos={0 0}, upload=526987]: Long tail cancellation due to optimal threshold
Dec 30 16:44:59.009857671 (601h15m58.880940706s) [noise=true , address=145.79.227.33:4903 , piece_id=THSFTSO2Y4QVWIGNUOU4AKFGTSQWQ6FLNIIDISIVNRF7XZZWWPCQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:59.009921232 (601h15m58.881004267s) [noise=true , address=145.79.227.30:4790 , piece_id=5VRI4FYK7J54TCOY7DX6TCQ3634N3TATGDZUZDWRBBLMXCGRVR3Q, seg_pos={0 0}, upload=526987]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kuSMof8EbvsaMCdVxKWjgTT5TfLJnTAeHUf8DKNC3UGQn5q7T): protocol: expected piece hash; context canceled; EOF
Dec 30 16:44:59.009927314 (601h15m58.881010359s) [noise=true , address=145.79.227.30:4790 , piece_id=5VRI4FYK7J54TCOY7DX6TCQ3634N3TATGDZUZDWRBBLMXCGRVR3Q, seg_pos={0 0}, upload=526987]: Long tail cancellation due to optimal threshold
Dec 30 16:44:59.010092589 (601h15m58.881175624s) [noise=true , address=77.90.31.11:4567 , piece_id=NMCHY46TZ6W6EPVLVVCGQXHFERHBLSFB3J55AD32KQSKOVXABO7A, seg_pos={0 0}, upload=526987]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HJVmEuxjrwWCcdwxWYFC5fTULds1kiRH82DiRWipWhRT1Qwdk): protocol: expected piece hash; context canceled; EOF
Dec 30 16:44:59.010101946 (601h15m58.881184981s) [noise=true , address=77.90.31.11:4567 , piece_id=NMCHY46TZ6W6EPVLVVCGQXHFERHBLSFB3J55AD32KQSKOVXABO7A, seg_pos={0 0}, upload=526987]: Long tail cancellation due to optimal threshold
Dec 30 16:44:59.010125410 (601h15m58.881208445s) [seg_pos={0 0}, upload=526987]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil>
Dec 30 16:44:59.010204091 (601h15m58.881287136s) [upload=526987]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 16:44:59.010211274 (601h15m58.881294339s) [upload=526987]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 16:44:59.309660956 (601h15m59.180744001s) [upload=526987]: Done uploading segments...
Dec 30 16:44:59.309679662 (601h15m59.180762697s) [upload=526987]: Upload finished. err: <nil>
Duration: 34.341346745s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
25 out of 60 is failed (threshold=3) [XXXX........X..X....X.XXX..XX.X.X..XXX.XX..X.X.......X.X..XX] duration:
30.004049642s trace:
https://jaeger.dev.storj.io/trace/411e3d14de7b3865
FAIL (2025-12-30 17:03:47 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 30 17:03:18.654418548 (601h34m18.525501583s) [upload=527254]: Starting upload
Dec 30 17:03:18.654472180 (601h34m18.525555215s) [upload=527254]: Done starting upload
Dec 30 17:03:18.654572972 (601h34m18.525656017s) [upload=527254]: Uploading segments...
Dec 30 17:03:18.654725121 (601h34m18.525808156s) [upload=527254]: Got next segment. Inline: false
Dec 30 17:03:18.654775547 (601h34m18.525858583s) [upload=527254]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 17:03:18.654781308 (601h34m18.525864353s) [upload=527254]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 17:03:18.901136047 (601h34m18.772219102s) [seg_pos={0 0}, upload=527254]: Begin upload segment...
Dec 30 17:03:18.901196763 (601h34m18.772279808s) [seg_pos={0 0}, upload=527254]: Done begin upload segment.
Dec 30 17:03:18.901226700 (601h34m18.772309735s) [noise=true , address=45.91.116.10:4081 , piece_id=4Z5V7GUGUH3EIQ7TQTUMW7T3Z4GYZHKABTP3G5Z2KXLX4NXHZO2A, seg_pos={0 0}, upload=527254]: Uploading piece...
Dec 30 17:03:18.901309127 (601h34m18.772392172s) [noise=true , address=145.79.227.17:4970 , piece_id=SHRTIGRS6KK6PW3TD2JOVMFHD7ZKDQ2IMOGZB4QYBZTMMJL6XMEQ, seg_pos={0 0}, upload=527254]: Uploading piece...
Dec 30 17:03:18.901758002 (601h34m18.772841037s) [noise=true , address=145.79.227.3:4149 , piece_id=LNIGGIEGL2F3SCYTCCOEZKBSBEV3GSYFGWEHPGP7UWBANW5VPQCQ, seg_pos={0 0}, upload=527254]: Uploading piece...
Dec 30 17:03:18.902077350 (601h34m18.773160385s) [noise=true , address=145.79.227.50:4373 , piece_id=N6GITFLLTPU7A32DIPBGVAAPAH6AJE6E2NTMR4S4SXGIUECLAN2Q, seg_pos={0 0}, upload=527254]: Uploading piece...
Dec 30 17:03:18.902360430 (601h34m18.773443465s) [noise=true , address=145.79.227.15:4876 , piece_id=V2OXOBNCJYYLVA3WW55BXZ5VK6LS2NVKYYLPL46O7TH53H4VEBMA, seg_pos={0 0}, upload=527254]: Uploading piece...
Dec 30 17:03:18.902708793 (601h34m18.773791828s) [noise=true , address=77.90.31.18:4805 , piece_id=7DCRBKUNZJ2DSDS26WRNRV6JTHBZA23A4L6S5VSRU7PMASBKHSXA, seg_pos={0 0}, upload=527254]: Uploading piece...
Dec 30 17:03:18.903009074 (601h34m18.774092109s) [noise=true , address=145.79.227.45:4230 , piece_id=QTBZMM7G37DLTHPVBROH37RRXQYHT7PQB4AQQ7H6TXSTTE5XFMNQ, seg_pos={0 0}, upload=527254]: Uploading piece...
Dec 30 17:03:18.903293646 (601h34m18.774376681s) [noise=true , address=145.79.227.23:4545 , piece_id=4O6USJ7XQ3SSYCIV5BZ6AIYQLSFAIROYLEHGUDDVZQDOSREL4NUA, seg_pos={0 0}, upload=527254]: Uploading piece...
Dec 30 17:03:18.903640626 (601h34m18.774723662s) [noise=true , address=145.79.227.48:4302 , piece_id=JLPS26KBBMNG6ZSNMOYZPUF5OLYE64JUZRTQQBD7BQCC4ABJTMSQ, seg_pos={0 0}, upload=527254]: Uploading piece...
Dec 30 17:03:18.905010977 (601h34m18.776094012s) [noise=true , address=145.79.227.10:4402 , piece_id=D2EOI6P3G3REU4E5EIOINVUIXWCLIL7W7UOORJBPXVE6347EIZQA, seg_pos={0 0}, upload=527254]: Uploading piece...
Dec 30 17:03:18.906509580 (601h34m18.777592615s) [noise=true , address=145.79.227.24:4581 , piece_id=DU2BRTNM2GJVG2RY2BI3Y53342UW3SE7BC6UDX7Y4HRSHCYXUVCA, seg_pos={0 0}, upload=527254]: Uploading piece...
Dec 30 17:03:18.907799827 (601h34m18.778882862s) [noise=true , address=77.90.31.12:4589 , piece_id=ZPQWP76WNPDCG5LV3TFZJDWSJSMJKPJSKBRBZQQ3NGY2JK7WSANA, seg_pos={0 0}, upload=527254]: Uploading piece...
Dec 30 17:03:18.909107506 (601h34m18.780190541s) [noise=true , address=45.91.116.4:4937 , piece_id=2IN6YCNOFSZE4SO5A7FH4662UFBTSKKAKWP7VUIEZQXNME4GQY3A, seg_pos={0 0}, upload=527254]: Uploading piece...
Dec 30 17:03:18.910473167 (601h34m18.781556202s) [noise=true , address=145.79.227.25:4617 , piece_id=4QAQOAFIYOMPZPMBUKMSNHZDLMSSUP6XTAHUEIJOG7IRXOD2764A, seg_pos={0 0}, upload=527254]: Uploading piece...
Dec 30 17:03:18.911857843 (601h34m18.782940878s) [noise=true , address=145.79.227.4:4185 , piece_id=RGJDDGO7HLKKIKG2KM5ITAWTXSOWCUO4PC6QTZNQ63MNYIAFTURA, seg_pos={0 0}, upload=527254]: Uploading piece...
Dec 30 17:03:18.912354930 (601h34m18.783437975s) [noise=true , address=77.90.31.10:4517 , piece_id=XGYFOBF3RWEDX4HDLDJU4SSYWQ3RLXGSQVJNKF5VCU777CADCVCA, seg_pos={0 0}, upload=527254]: Uploading piece...
Dec 30 17:03:20.603031825 (601h34m20.47411487s) [upload=527254]: Next returned nil segment
Dec 30 17:03:20.603079595 (601h34m20.474162631s) [upload=527254]: Waiting for error group managing segments...
Dec 30 17:03:20.882104073 (601h34m20.753187108s) [noise=true , address=145.79.227.50:4373 , piece_id=N6GITFLLTPU7A32DIPBGVAAPAH6AJE6E2NTMR4S4SXGIUECLAN2Q, seg_pos={0 0}, upload=527254]: Done uploading piece. err: <nil>
Dec 30 17:03:20.888923718 (601h34m20.760006763s) [noise=true , address=145.79.227.23:4545 , piece_id=4O6USJ7XQ3SSYCIV5BZ6AIYQLSFAIROYLEHGUDDVZQDOSREL4NUA, seg_pos={0 0}, upload=527254]: Done uploading piece. err: <nil>
Dec 30 17:03:20.894580499 (601h34m20.765663544s) [noise=true , address=145.79.227.3:4149 , piece_id=LNIGGIEGL2F3SCYTCCOEZKBSBEV3GSYFGWEHPGP7UWBANW5VPQCQ, seg_pos={0 0}, upload=527254]: Done uploading piece. err: <nil>
Dec 30 17:03:20.896608281 (601h34m20.767691326s) [noise=true , address=145.79.227.15:4876 , piece_id=V2OXOBNCJYYLVA3WW55BXZ5VK6LS2NVKYYLPL46O7TH53H4VEBMA, seg_pos={0 0}, upload=527254]: Done uploading piece. err: <nil>
Dec 30 17:03:20.899132577 (601h34m20.770215622s) [noise=true , address=145.79.227.17:4970 , piece_id=SHRTIGRS6KK6PW3TD2JOVMFHD7ZKDQ2IMOGZB4QYBZTMMJL6XMEQ, seg_pos={0 0}, upload=527254]: Done uploading piece. err: <nil>
Dec 30 17:03:20.901263484 (601h34m20.772346529s) [noise=true , address=145.79.227.24:4581 , piece_id=DU2BRTNM2GJVG2RY2BI3Y53342UW3SE7BC6UDX7Y4HRSHCYXUVCA, seg_pos={0 0}, upload=527254]: Done uploading piece. err: <nil>
Dec 30 17:03:20.908995056 (601h34m20.780078091s) [noise=true , address=145.79.227.10:4402 , piece_id=D2EOI6P3G3REU4E5EIOINVUIXWCLIL7W7UOORJBPXVE6347EIZQA, seg_pos={0 0}, upload=527254]: Done uploading piece. err: <nil>
Dec 30 17:03:20.911370389 (601h34m20.782453434s) [noise=true , address=145.79.227.4:4185 , piece_id=RGJDDGO7HLKKIKG2KM5ITAWTXSOWCUO4PC6QTZNQ63MNYIAFTURA, seg_pos={0 0}, upload=527254]: Done uploading piece. err: <nil>
Dec 30 17:03:20.917279470 (601h34m20.788362505s) [noise=true , address=145.79.227.48:4302 , piece_id=JLPS26KBBMNG6ZSNMOYZPUF5OLYE64JUZRTQQBD7BQCC4ABJTMSQ, seg_pos={0 0}, upload=527254]: Done uploading piece. err: <nil>
Dec 30 17:03:20.920107986 (601h34m20.791191031s) [noise=true , address=145.79.227.45:4230 , piece_id=QTBZMM7G37DLTHPVBROH37RRXQYHT7PQB4AQQ7H6TXSTTE5XFMNQ, seg_pos={0 0}, upload=527254]: Done uploading piece. err: <nil>
Dec 30 17:03:20.944777200 (601h34m20.815860235s) [noise=true , address=145.79.227.25:4617 , piece_id=4QAQOAFIYOMPZPMBUKMSNHZDLMSSUP6XTAHUEIJOG7IRXOD2764A, seg_pos={0 0}, upload=527254]: Done uploading piece. err: <nil>
Dec 30 17:03:21.095378983 (601h34m20.966462028s) [noise=true , address=77.90.31.12:4589 , piece_id=ZPQWP76WNPDCG5LV3TFZJDWSJSMJKPJSKBRBZQQ3NGY2JK7WSANA, seg_pos={0 0}, upload=527254]: Done uploading piece. err: <nil>
Dec 30 17:03:21.476737317 (601h34m21.347820352s) [noise=true , address=77.90.31.10:4517 , piece_id=XGYFOBF3RWEDX4HDLDJU4SSYWQ3RLXGSQVJNKF5VCU777CADCVCA, seg_pos={0 0}, upload=527254]: Done uploading piece. err: <nil>
Dec 30 17:03:21.476766633 (601h34m21.347849668s) [seg_pos={0 0}, upload=527254]: Segment reached optimal threshold of 13 pieces.
Dec 30 17:03:21.476995379 (601h34m21.348078424s) [noise=true , address=45.91.116.10:4081 , piece_id=4Z5V7GUGUH3EIQ7TQTUMW7T3Z4GYZHKABTP3G5Z2KXLX4NXHZO2A, seg_pos={0 0}, upload=527254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: write tcp 188.166.225.170:35034->45.91.116.10:4081: use of closed network connection; write tcp 188.166.225.170:35034->45.91.116.10:4081: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35034->45.91.116.10:4081: use of closed network connection
Dec 30 17:03:21.477004907 (601h34m21.348087942s) [noise=true , address=45.91.116.10:4081 , piece_id=4Z5V7GUGUH3EIQ7TQTUMW7T3Z4GYZHKABTP3G5Z2KXLX4NXHZO2A, seg_pos={0 0}, upload=527254]: Long tail cancellation due to optimal threshold
Dec 30 17:03:21.477175773 (601h34m21.348258808s) [noise=true , address=77.90.31.18:4805 , piece_id=7DCRBKUNZJ2DSDS26WRNRV6JTHBZA23A4L6S5VSRU7PMASBKHSXA, seg_pos={0 0}, upload=527254]: Done uploading piece. err: <nil>
Dec 30 17:03:21.477360594 (601h34m21.348443629s) [noise=true , address=45.91.116.4:4937 , piece_id=2IN6YCNOFSZE4SO5A7FH4662UFBTSKKAKWP7VUIEZQXNME4GQY3A, seg_pos={0 0}, upload=527254]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ry1jh9FXr14g7WU4bfgJH4qw71kmTugE5qbqEUZmf7oVZYRkq): protocol: expected piece hash; context canceled; EOF
Dec 30 17:03:21.477371144 (601h34m21.348454189s) [noise=true , address=45.91.116.4:4937 , piece_id=2IN6YCNOFSZE4SO5A7FH4662UFBTSKKAKWP7VUIEZQXNME4GQY3A, seg_pos={0 0}, upload=527254]: Long tail cancellation due to optimal threshold
Dec 30 17:03:21.477392435 (601h34m21.34847547s) [seg_pos={0 0}, upload=527254]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil>
Dec 30 17:03:21.477497255 (601h34m21.3485803s) [upload=527254]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 17:03:21.477506582 (601h34m21.348589637s) [upload=527254]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 17:03:21.761466016 (601h34m21.632549071s) [upload=527254]: Done uploading segments...
Dec 30 17:03:21.761492797 (601h34m21.632575842s) [upload=527254]: Upload finished. err: <nil>
Duration: 30.004065834s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
20 out of 60 is failed (threshold=6) [XXX.X.X.X...XXX..X....XXX..X...XX......X.XX................X] duration:
5.00122322s trace:
https://jaeger.dev.storj.io/trace/16a0763f8dcf089e
FAIL (2025-12-30 17:03:30 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001247024s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [..............X.............X.......X.......................] duration:
37.513988219s trace:
https://jaeger.dev.storj.io/trace/3f4379d3637c8e4b
FAIL (2025-12-30 17:03:04 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1WBuBdGxdPXi6VsNf4FJpn3LpWoQztM7CV6YjAW1mUDVjAUjn2, Piece ID: MS4HNXUQIITLBMEHZDPJMEH7FSLMBBGKJSIP5BY2BDUF3IE2NKIQ) context deadline exceeded; piecestore: (Node ID: 1soiTQsiMNe5BkNHwwAGykmtP9LtLxncqE7Tv728ak8pjFeshg, Piece ID: BHVQNSBU3LP35Z5WQYTRPJZNNWTDZKWYK2CMRLHSOGBJ4IXGZ5SA) context deadline exceeded; piecestore: (Node ID: 1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, Piece ID: MVRLWV5URMPV57R52J426SXY5ZWQGHOTL42GO2GPEZ4LA7B2JGCQ) context deadline exceeded
======LOGS======
5m56.099625936s) [seg_pos={0 0}, upload=526987]: Begin upload segment...
Dec 30 16:44:56.228605921 (601h15m56.099688967s) [seg_pos={0 0}, upload=526987]: Done begin upload segment.
Dec 30 16:44:56.228736641 (601h15m56.099819686s) [noise=true , address=145.79.227.34:4923 , piece_id=OB3KQEHYXE4BL73EDDQDN2LSVYJYYQQFUSFJ655BFJISLBGOH4XQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.228753061 (601h15m56.099836117s) [noise=true , address=64.49.13.26:4947 , piece_id=BHVQNSBU3LP35Z5WQYTRPJZNNWTDZKWYK2CMRLHSOGBJ4IXGZ5SA, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.229701116 (601h15m56.100784151s) [noise=true , address=45.91.116.8:4071 , piece_id=3PUQCDJAXSQGEM47BTD2B5HWWNK3PUS6AEVQ63BEC2Q2CKCB4PXQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.230399837 (601h15m56.101482882s) [noise=true , address=77.90.31.21:4899 , piece_id=DCYPDKTC4S5LG3XEBQWZT3HGFJ63T3UC3PFUDLF7OOJHNMYW4KRQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.230834926 (601h15m56.101917971s) [noise=true , address=77.90.31.19:4853 , piece_id=MS4HNXUQIITLBMEHZDPJMEH7FSLMBBGKJSIP5BY2BDUF3IE2NKIQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.232511007 (601h15m56.103594112s) [noise=true , address=145.79.227.8:4344 , piece_id=6L5DD7CQJ5JZ7IPFX2IGHNEP7M6BZU2DA5Z5GEOKGPAMVZUXQHOQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.233047428 (601h15m56.104130473s) [noise=true , address=145.79.227.31:4808 , piece_id=MW4UFSJ4XFLXK7ZLWZFWTP2N4QQUS24WP3O44DJWT7AK2WCWAT3Q, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.233557500 (601h15m56.104640545s) [noise=true , address=145.79.227.33:4903 , piece_id=THSFTSO2Y4QVWIGNUOU4AKFGTSQWQ6FLNIIDISIVNRF7XZZWWPCQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.234065937 (601h15m56.105148983s) [noise=true , address=64.49.13.137:4493 , piece_id=QZSPIHMAGKSHSAJYJ6BR26EMTCWFAEPOPC4O65YN6SL6222H4HWQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.234513860 (601h15m56.105596896s) [noise=true , address=64.49.13.27:4965 , piece_id=AQTJD7BXKPMIAGEOGJCY5Y64IVU2SHCZEFJL2NLQAHVX5HR33N7Q, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.236125519 (601h15m56.107208554s) [noise=true , address=77.90.31.11:4567 , piece_id=NMCHY46TZ6W6EPVLVVCGQXHFERHBLSFB3J55AD32KQSKOVXABO7A, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.237179756 (601h15m56.108262791s) [noise=true , address=145.79.227.40:4064 , piece_id=SEYSDBZHX6SHMFZBGY3XDIYEOTWFCNGG2KNNTCIXYQRZP7FZOVJQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.238439144 (601h15m56.109522189s) [noise=true , address=145.79.227.30:4790 , piece_id=5VRI4FYK7J54TCOY7DX6TCQ3634N3TATGDZUZDWRBBLMXCGRVR3Q, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.239316585 (601h15m56.11039962s) [noise=true , address=145.79.227.44:4212 , piece_id=ZV3XMTVGJR2D5X4EEGGDTGGGXSQI4QNCJZCI3GPR6RTD7JEQWLGA, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.242138468 (601h15m56.113221503s) [noise=true , address=45.91.116.11:4131 , piece_id=YAFIDLBM762GWTNKDW62FMNLE4YHCNYW2WU3R4AMO5NELMVOHHHQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.243965547 (601h15m56.115048582s) [noise=true , address=77.90.31.18:4829 , piece_id=MVRLWV5URMPV57R52J426SXY5ZWQGHOTL42GO2GPEZ4LA7B2JGCQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.250805150 (601h15m56.121888185s) [noise=true , address=5.231.73.4:4033 , piece_id=CCH6AH47YBQPGPF4MPZU2NPT7HPF4D3KT3ZVDITRYF7UW5JUKWDQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.254973396 (601h15m56.126056431s) [noise=true , address=145.79.227.49:4341 , piece_id=6GKQQIBOECLDEEDZNO5AYKX7XMXZYUA6WNGKD3FQSF2Q4O6A4MQA, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:57.691193696 (601h15m57.562276731s) [upload=526987]: Next returned nil segment
Dec 30 16:44:57.691206460 (601h15m57.562289495s) [upload=526987]: Waiting for error group managing segments...
Dec 30 16:44:58.085111563 (601h15m57.956194598s) [noise=true , address=64.49.13.137:4493 , piece_id=QZSPIHMAGKSHSAJYJ6BR26EMTCWFAEPOPC4O65YN6SL6222H4HWQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.086614434 (601h15m57.957697479s) [noise=true , address=64.49.13.27:4965 , piece_id=AQTJD7BXKPMIAGEOGJCY5Y64IVU2SHCZEFJL2NLQAHVX5HR33N7Q, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.119421281 (601h15m57.990504316s) [noise=true , address=145.79.227.8:4344 , piece_id=6L5DD7CQJ5JZ7IPFX2IGHNEP7M6BZU2DA5Z5GEOKGPAMVZUXQHOQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.125317667 (601h15m57.996400702s) [noise=true , address=145.79.227.31:4808 , piece_id=MW4UFSJ4XFLXK7ZLWZFWTP2N4QQUS24WP3O44DJWT7AK2WCWAT3Q, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.262371457 (601h15m58.133454502s) [noise=true , address=64.49.13.26:4947 , piece_id=BHVQNSBU3LP35Z5WQYTRPJZNNWTDZKWYK2CMRLHSOGBJ4IXGZ5SA, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.284980216 (601h15m58.156063251s) [noise=true , address=77.90.31.19:4853 , piece_id=MS4HNXUQIITLBMEHZDPJMEH7FSLMBBGKJSIP5BY2BDUF3IE2NKIQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.387549123 (601h15m58.258632158s) [noise=true , address=77.90.31.18:4829 , piece_id=MVRLWV5URMPV57R52J426SXY5ZWQGHOTL42GO2GPEZ4LA7B2JGCQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.394280961 (601h15m58.265364006s) [noise=true , address=5.231.73.4:4033 , piece_id=CCH6AH47YBQPGPF4MPZU2NPT7HPF4D3KT3ZVDITRYF7UW5JUKWDQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.510328564 (601h15m58.381411609s) [noise=true , address=45.91.116.8:4071 , piece_id=3PUQCDJAXSQGEM47BTD2B5HWWNK3PUS6AEVQ63BEC2Q2CKCB4PXQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.545484694 (601h15m58.416567739s) [noise=true , address=45.91.116.11:4131 , piece_id=YAFIDLBM762GWTNKDW62FMNLE4YHCNYW2WU3R4AMO5NELMVOHHHQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.710409562 (601h15m58.581492607s) [noise=true , address=77.90.31.21:4899 , piece_id=DCYPDKTC4S5LG3XEBQWZT3HGFJ63T3UC3PFUDLF7OOJHNMYW4KRQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.981893937 (601h15m58.852976982s) [noise=true , address=145.79.227.44:4212 , piece_id=ZV3XMTVGJR2D5X4EEGGDTGGGXSQI4QNCJZCI3GPR6RTD7JEQWLGA, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.995663946 (601h15m58.866746991s) [noise=true , address=145.79.227.49:4341 , piece_id=6GKQQIBOECLDEEDZNO5AYKX7XMXZYUA6WNGKD3FQSF2Q4O6A4MQA, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:59.009469091 (601h15m58.880552136s) [noise=true , address=145.79.227.40:4064 , piece_id=SEYSDBZHX6SHMFZBGY3XDIYEOTWFCNGG2KNNTCIXYQRZP7FZOVJQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:59.009489950 (601h15m58.880572986s) [seg_pos={0 0}, upload=526987]: Segment reached optimal threshold of 14 pieces.
Dec 30 16:44:59.009662329 (601h15m58.880745374s) [noise=true , address=145.79.227.34:4923 , piece_id=OB3KQEHYXE4BL73EDDQDN2LSVYJYYQQFUSFJ655BFJISLBGOH4XQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VXAxRGiJQor24W7DYSeUQUu1QF7EcX6BQYgfUheACRs2quFFu): protocol: expected piece hash; context canceled; EOF
Dec 30 16:44:59.009668390 (601h15m58.880751425s) [noise=true , address=145.79.227.34:4923 , piece_id=OB3KQEHYXE4BL73EDDQDN2LSVYJYYQQFUSFJ655BFJISLBGOH4XQ, seg_pos={0 0}, upload=526987]: Long tail cancellation due to optimal threshold
Dec 30 16:44:59.009857671 (601h15m58.880940706s) [noise=true , address=145.79.227.33:4903 , piece_id=THSFTSO2Y4QVWIGNUOU4AKFGTSQWQ6FLNIIDISIVNRF7XZZWWPCQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:59.009921232 (601h15m58.881004267s) [noise=true , address=145.79.227.30:4790 , piece_id=5VRI4FYK7J54TCOY7DX6TCQ3634N3TATGDZUZDWRBBLMXCGRVR3Q, seg_pos={0 0}, upload=526987]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kuSMof8EbvsaMCdVxKWjgTT5TfLJnTAeHUf8DKNC3UGQn5q7T): protocol: expected piece hash; context canceled; EOF
Dec 30 16:44:59.009927314 (601h15m58.881010359s) [noise=true , address=145.79.227.30:4790 , piece_id=5VRI4FYK7J54TCOY7DX6TCQ3634N3TATGDZUZDWRBBLMXCGRVR3Q, seg_pos={0 0}, upload=526987]: Long tail cancellation due to optimal threshold
Dec 30 16:44:59.010092589 (601h15m58.881175624s) [noise=true , address=77.90.31.11:4567 , piece_id=NMCHY46TZ6W6EPVLVVCGQXHFERHBLSFB3J55AD32KQSKOVXABO7A, seg_pos={0 0}, upload=526987]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HJVmEuxjrwWCcdwxWYFC5fTULds1kiRH82DiRWipWhRT1Qwdk): protocol: expected piece hash; context canceled; EOF
Dec 30 16:44:59.010101946 (601h15m58.881184981s) [noise=true , address=77.90.31.11:4567 , piece_id=NMCHY46TZ6W6EPVLVVCGQXHFERHBLSFB3J55AD32KQSKOVXABO7A, seg_pos={0 0}, upload=526987]: Long tail cancellation due to optimal threshold
Dec 30 16:44:59.010125410 (601h15m58.881208445s) [seg_pos={0 0}, upload=526987]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil>
Dec 30 16:44:59.010204091 (601h15m58.881287136s) [upload=526987]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 16:44:59.010211274 (601h15m58.881294339s) [upload=526987]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 16:44:59.309660956 (601h15m59.180744001s) [upload=526987]: Done uploading segments...
Dec 30 16:44:59.309679662 (601h15m59.180762697s) [upload=526987]: Upload finished. err: <nil>
Duration: 37.514016713s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
24 out of 60 is failed (threshold=3) [.XXXX........X..X....X.XXX..XX.X.X..XXX.XX..X.X.......X.X..X] duration:
21.872290404s trace:
https://jaeger.dev.storj.io/trace/7963a6fad29dd7a
FAIL (2025-12-30 17:02:53 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
Dec 30 17:02:32.188456576 (601h33m32.059539611s) [upload=527240]: Starting upload
Dec 30 17:02:32.188510329 (601h33m32.059593364s) [upload=527240]: Done starting upload
Dec 30 17:02:32.188684400 (601h33m32.059767445s) [upload=527240]: Uploading segments...
Dec 30 17:02:32.188795993 (601h33m32.059879038s) [upload=527240]: Got next segment. Inline: false
Dec 30 17:02:32.188853111 (601h33m32.059936156s) [upload=527240]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 17:02:32.188859033 (601h33m32.059942078s) [upload=527240]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 17:02:32.435006227 (601h33m32.306089262s) [seg_pos={0 0}, upload=527240]: Begin upload segment...
Dec 30 17:02:32.435067372 (601h33m32.306150407s) [seg_pos={0 0}, upload=527240]: Done begin upload segment.
Dec 30 17:02:32.435098902 (601h33m32.306181938s) [noise=true , address=145.79.227.51:4411 , piece_id=NJCY4JSVAOAEO22S7K3EE2U3ON6PYGKMFD6CADM5X2SBNOG35ESA, seg_pos={0 0}, upload=527240]: Uploading piece...
Dec 30 17:02:32.435169807 (601h33m32.306252873s) [noise=true , address=45.91.116.14:4192 , piece_id=VMZQGYE7EZTGUMQWC5AKKXBQ2ILZAWIM7VLNYJED5LGACUJV3C6Q, seg_pos={0 0}, upload=527240]: Uploading piece...
Dec 30 17:02:32.435679088 (601h33m32.306762123s) [noise=true , address=145.79.227.8:4330 , piece_id=25C7IWOT6AAX7RIDEC2EMT5J4YSICD677X7K3GDFEN2PSPLENCJQ, seg_pos={0 0}, upload=527240]: Uploading piece...
Dec 30 17:02:32.436029054 (601h33m32.307112089s) [noise=true , address=145.79.227.33:4871 , piece_id=EKYLVHQPWLS5MV6NQNC7FQUJCL7PB2AV6EVZMTM5QTEAOC5PKZ6Q, seg_pos={0 0}, upload=527240]: Uploading piece...
Dec 30 17:02:32.436048221 (601h33m32.307131256s) [noise=true , address=77.90.31.12:4590 , piece_id=35YFHKJZJKLYBE63D2D3XJCQ74D7EUZGDRM67X55WTEXNQP7JTMA, seg_pos={0 0}, upload=527240]: Uploading piece...
Dec 30 17:02:32.436295462 (601h33m32.307378497s) [noise=true , address=145.79.227.9:4366 , piece_id=T6WPAMGNUXOSBWZWACCI3VVDDFXHNDOCBKA7FMX3GAB2A56ISLJA, seg_pos={0 0}, upload=527240]: Uploading piece...
Dec 30 17:02:32.436613397 (601h33m32.307696432s) [noise=true , address=145.79.227.48:4301 , piece_id=AMDSTPBWZPQOEOHWFGRXKOFQQAF4JNM76OTTOIXHPUB3WMJXIUHA, seg_pos={0 0}, upload=527240]: Uploading piece...
Dec 30 17:02:32.436965848 (601h33m32.308048883s) [noise=true , address=45.91.116.6:4974 , piece_id=C7QYKUSMHC3HU2RJA2MPF7UDOQTCR5QSQS6UL4DZZPZ4TNWBZBQA, seg_pos={0 0}, upload=527240]: Uploading piece...
Dec 30 17:02:32.437007427 (601h33m32.308090462s) [noise=true , address=145.79.227.43:4159 , piece_id=OPTHNDUG5UG3T5GN7A4FBIVJ2NTSGREWT2YNZO4B6DIK3S2PMLIQ, seg_pos={0 0}, upload=527240]: Uploading piece...
Dec 30 17:02:32.437275728 (601h33m32.308358763s) [noise=true , address=145.79.227.4:4185 , piece_id=BJDDCCKXEP4EKZX6HQUENGB6W5D7GCWPELWKZJPSF45VE3ENQAKQ, seg_pos={0 0}, upload=527240]: Uploading piece...
Dec 30 17:02:32.437319832 (601h33m32.308402877s) [noise=true , address=45.91.116.13:4156 , piece_id=MUVZCP35R6SLZG5SYKQFHZGO5PG6UVUCTMKCH7UDGSHJOUNN7GKQ, seg_pos={0 0}, upload=527240]: Uploading piece...
Dec 30 17:02:32.437602961 (601h33m32.308685996s) [noise=true , address=45.91.116.7:4010 , piece_id=ZTT4P25OCUJSQ66UVQPWN4XR2RE5YVHETKKH2MPZFW6Z2UWGF4OQ, seg_pos={0 0}, upload=527240]: Uploading piece...
Dec 30 17:02:32.437873135 (601h33m32.30895618s) [noise=true , address=145.79.227.47:4267 , piece_id=TAZG7EZHP7JXIHPXQUJ2NY5JAANKQ5ROYJGC2KCTSOM3J5I2R4GQ, seg_pos={0 0}, upload=527240]: Uploading piece...
Dec 30 17:02:32.438111229 (601h33m32.309194264s) [noise=true , address=145.79.227.50:4373 , piece_id=RJR3MYJQOLRCBG4EPVGER2WU2CBWMLDEJVILZDRENPTYGAQ467JA, seg_pos={0 0}, upload=527240]: Uploading piece...
Dec 30 17:02:32.438367366 (601h33m32.309450401s) [noise=true , address=77.90.31.16:4736 , piece_id=2DWZ5UIQFCMVZARHJ4EJ22O5WG5L6KUMK3EELBJ3A3TMZGPBRG2A, seg_pos={0 0}, upload=527240]: Uploading piece...
Dec 30 17:02:32.438629235 (601h33m32.30971227s) [noise=true , address=145.79.227.19:4078 , piece_id=M2ODK3ETFSKVPFIXJFWCUC4UI3GVPLLE43TODZE4WDII5MY4ZI6A, seg_pos={0 0}, upload=527240]: Uploading piece...
Dec 30 17:02:34.134719898 (601h33m34.005802933s) [upload=527240]: Next returned nil segment
Dec 30 17:02:34.134729736 (601h33m34.005812771s) [upload=527240]: Waiting for error group managing segments...
Dec 30 17:02:34.424592330 (601h33m34.295675365s) [noise=true , address=145.79.227.51:4411 , piece_id=NJCY4JSVAOAEO22S7K3EE2U3ON6PYGKMFD6CADM5X2SBNOG35ESA, seg_pos={0 0}, upload=527240]: Done uploading piece. err: <nil>
Dec 30 17:02:34.445050766 (601h33m34.316133811s) [noise=true , address=145.79.227.8:4330 , piece_id=25C7IWOT6AAX7RIDEC2EMT5J4YSICD677X7K3GDFEN2PSPLENCJQ, seg_pos={0 0}, upload=527240]: Done uploading piece. err: <nil>
Dec 30 17:02:34.447508435 (601h33m34.31859148s) [noise=true , address=145.79.227.4:4185 , piece_id=BJDDCCKXEP4EKZX6HQUENGB6W5D7GCWPELWKZJPSF45VE3ENQAKQ, seg_pos={0 0}, upload=527240]: Done uploading piece. err: <nil>
Dec 30 17:02:34.452249753 (601h33m34.323332798s) [noise=true , address=145.79.227.50:4373 , piece_id=RJR3MYJQOLRCBG4EPVGER2WU2CBWMLDEJVILZDRENPTYGAQ467JA, seg_pos={0 0}, upload=527240]: Done uploading piece. err: <nil>
Dec 30 17:02:34.454323912 (601h33m34.325406947s) [noise=true , address=145.79.227.48:4301 , piece_id=AMDSTPBWZPQOEOHWFGRXKOFQQAF4JNM76OTTOIXHPUB3WMJXIUHA, seg_pos={0 0}, upload=527240]: Done uploading piece. err: <nil>
Dec 30 17:02:34.455394541 (601h33m34.326477576s) [noise=true , address=145.79.227.47:4267 , piece_id=TAZG7EZHP7JXIHPXQUJ2NY5JAANKQ5ROYJGC2KCTSOM3J5I2R4GQ, seg_pos={0 0}, upload=527240]: Done uploading piece. err: <nil>
Dec 30 17:02:34.463495696 (601h33m34.334578731s) [noise=true , address=145.79.227.19:4078 , piece_id=M2ODK3ETFSKVPFIXJFWCUC4UI3GVPLLE43TODZE4WDII5MY4ZI6A, seg_pos={0 0}, upload=527240]: Done uploading piece. err: <nil>
Dec 30 17:02:34.472512415 (601h33m34.34359545s) [noise=true , address=145.79.227.9:4366 , piece_id=T6WPAMGNUXOSBWZWACCI3VVDDFXHNDOCBKA7FMX3GAB2A56ISLJA, seg_pos={0 0}, upload=527240]: Done uploading piece. err: <nil>
Dec 30 17:02:34.610986978 (601h33m34.482070013s) [noise=true , address=77.90.31.16:4736 , piece_id=2DWZ5UIQFCMVZARHJ4EJ22O5WG5L6KUMK3EELBJ3A3TMZGPBRG2A, seg_pos={0 0}, upload=527240]: Done uploading piece. err: <nil>
Dec 30 17:02:34.701210254 (601h33m34.572293299s) [noise=true , address=77.90.31.12:4590 , piece_id=35YFHKJZJKLYBE63D2D3XJCQ74D7EUZGDRM67X55WTEXNQP7JTMA, seg_pos={0 0}, upload=527240]: Done uploading piece. err: <nil>
Dec 30 17:02:34.737555149 (601h33m34.608638194s) [noise=true , address=45.91.116.14:4192 , piece_id=VMZQGYE7EZTGUMQWC5AKKXBQ2ILZAWIM7VLNYJED5LGACUJV3C6Q, seg_pos={0 0}, upload=527240]: Done uploading piece. err: <nil>
Dec 30 17:02:34.927473386 (601h33m34.798556431s) [noise=true , address=45.91.116.13:4156 , piece_id=MUVZCP35R6SLZG5SYKQFHZGO5PG6UVUCTMKCH7UDGSHJOUNN7GKQ, seg_pos={0 0}, upload=527240]: Done uploading piece. err: <nil>
Dec 30 17:02:35.080246575 (601h33m34.95132961s) [noise=true , address=45.91.116.6:4974 , piece_id=C7QYKUSMHC3HU2RJA2MPF7UDOQTCR5QSQS6UL4DZZPZ4TNWBZBQA, seg_pos={0 0}, upload=527240]: Done uploading piece. err: <nil>
Dec 30 17:02:35.080263697 (601h33m34.951346742s) [seg_pos={0 0}, upload=527240]: Segment reached optimal threshold of 13 pieces.
Dec 30 17:02:35.080428742 (601h33m34.951511777s) [noise=true , address=145.79.227.33:4871 , piece_id=EKYLVHQPWLS5MV6NQNC7FQUJCL7PB2AV6EVZMTM5QTEAOC5PKZ6Q, seg_pos={0 0}, upload=527240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF
Dec 30 17:02:35.080435805 (601h33m34.95151884s) [noise=true , address=145.79.227.33:4871 , piece_id=EKYLVHQPWLS5MV6NQNC7FQUJCL7PB2AV6EVZMTM5QTEAOC5PKZ6Q, seg_pos={0 0}, upload=527240]: Long tail cancellation due to optimal threshold
Dec 30 17:02:35.080478676 (601h33m34.951561721s) [noise=true , address=145.79.227.43:4159 , piece_id=OPTHNDUG5UG3T5GN7A4FBIVJ2NTSGREWT2YNZO4B6DIK3S2PMLIQ, seg_pos={0 0}, upload=527240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULUPKjJwZocjV1R6Dy7vmWSrZ4hKx3YAHLAihgtMKf8Uw2UgA): protocol: expected piece hash; context canceled; EOF
Dec 30 17:02:35.080486762 (601h33m34.951569808s) [noise=true , address=145.79.227.43:4159 , piece_id=OPTHNDUG5UG3T5GN7A4FBIVJ2NTSGREWT2YNZO4B6DIK3S2PMLIQ, seg_pos={0 0}, upload=527240]: Long tail cancellation due to optimal threshold
Dec 30 17:02:35.080618142 (601h33m34.951701187s) [noise=true , address=45.91.116.7:4010 , piece_id=ZTT4P25OCUJSQ66UVQPWN4XR2RE5YVHETKKH2MPZFW6Z2UWGF4OQ, seg_pos={0 0}, upload=527240]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF
Dec 30 17:02:35.080655523 (601h33m34.951738558s) [noise=true , address=45.91.116.7:4010 , piece_id=ZTT4P25OCUJSQ66UVQPWN4XR2RE5YVHETKKH2MPZFW6Z2UWGF4OQ, seg_pos={0 0}, upload=527240]: Long tail cancellation due to optimal threshold
Dec 30 17:02:35.080673217 (601h33m34.951756252s) [seg_pos={0 0}, upload=527240]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 17:02:35.080785310 (601h33m34.951868345s) [upload=527240]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 17:02:35.080790710 (601h33m34.951873756s) [upload=527240]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 17:02:35.368484615 (601h33m35.23956766s) [upload=527240]: Done uploading segments...
Dec 30 17:02:35.368506958 (601h33m35.239590003s) [upload=527240]: Upload finished. err: <nil>
Duration: 21.872330732s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
19 out of 60 is failed (threshold=6) [.XXX.X.X.X...XXX..X....XXX..X...XX......X.XX................] duration:
3.583630438s trace:
https://jaeger.dev.storj.io/trace/2400bb34adb4564c
FAIL (2025-12-30 17:02:47 UTC)
orbiter warning: context deadline exceeded
Duration: 3.583659143s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [...............X.............X.......X......................] duration:
45.803786341s trace:
https://jaeger.dev.storj.io/trace/3f4379d3637c8e4b
FAIL (2025-12-30 17:02:21 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1WBuBdGxdPXi6VsNf4FJpn3LpWoQztM7CV6YjAW1mUDVjAUjn2, Piece ID: MS4HNXUQIITLBMEHZDPJMEH7FSLMBBGKJSIP5BY2BDUF3IE2NKIQ) context deadline exceeded; piecestore: (Node ID: 1soiTQsiMNe5BkNHwwAGykmtP9LtLxncqE7Tv728ak8pjFeshg, Piece ID: BHVQNSBU3LP35Z5WQYTRPJZNNWTDZKWYK2CMRLHSOGBJ4IXGZ5SA) context deadline exceeded; piecestore: (Node ID: 1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, Piece ID: MVRLWV5URMPV57R52J426SXY5ZWQGHOTL42GO2GPEZ4LA7B2JGCQ) context deadline exceeded
======LOGS======
5m56.099625936s) [seg_pos={0 0}, upload=526987]: Begin upload segment...
Dec 30 16:44:56.228605921 (601h15m56.099688967s) [seg_pos={0 0}, upload=526987]: Done begin upload segment.
Dec 30 16:44:56.228736641 (601h15m56.099819686s) [noise=true , address=145.79.227.34:4923 , piece_id=OB3KQEHYXE4BL73EDDQDN2LSVYJYYQQFUSFJ655BFJISLBGOH4XQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.228753061 (601h15m56.099836117s) [noise=true , address=64.49.13.26:4947 , piece_id=BHVQNSBU3LP35Z5WQYTRPJZNNWTDZKWYK2CMRLHSOGBJ4IXGZ5SA, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.229701116 (601h15m56.100784151s) [noise=true , address=45.91.116.8:4071 , piece_id=3PUQCDJAXSQGEM47BTD2B5HWWNK3PUS6AEVQ63BEC2Q2CKCB4PXQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.230399837 (601h15m56.101482882s) [noise=true , address=77.90.31.21:4899 , piece_id=DCYPDKTC4S5LG3XEBQWZT3HGFJ63T3UC3PFUDLF7OOJHNMYW4KRQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.230834926 (601h15m56.101917971s) [noise=true , address=77.90.31.19:4853 , piece_id=MS4HNXUQIITLBMEHZDPJMEH7FSLMBBGKJSIP5BY2BDUF3IE2NKIQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.232511007 (601h15m56.103594112s) [noise=true , address=145.79.227.8:4344 , piece_id=6L5DD7CQJ5JZ7IPFX2IGHNEP7M6BZU2DA5Z5GEOKGPAMVZUXQHOQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.233047428 (601h15m56.104130473s) [noise=true , address=145.79.227.31:4808 , piece_id=MW4UFSJ4XFLXK7ZLWZFWTP2N4QQUS24WP3O44DJWT7AK2WCWAT3Q, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.233557500 (601h15m56.104640545s) [noise=true , address=145.79.227.33:4903 , piece_id=THSFTSO2Y4QVWIGNUOU4AKFGTSQWQ6FLNIIDISIVNRF7XZZWWPCQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.234065937 (601h15m56.105148983s) [noise=true , address=64.49.13.137:4493 , piece_id=QZSPIHMAGKSHSAJYJ6BR26EMTCWFAEPOPC4O65YN6SL6222H4HWQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.234513860 (601h15m56.105596896s) [noise=true , address=64.49.13.27:4965 , piece_id=AQTJD7BXKPMIAGEOGJCY5Y64IVU2SHCZEFJL2NLQAHVX5HR33N7Q, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.236125519 (601h15m56.107208554s) [noise=true , address=77.90.31.11:4567 , piece_id=NMCHY46TZ6W6EPVLVVCGQXHFERHBLSFB3J55AD32KQSKOVXABO7A, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.237179756 (601h15m56.108262791s) [noise=true , address=145.79.227.40:4064 , piece_id=SEYSDBZHX6SHMFZBGY3XDIYEOTWFCNGG2KNNTCIXYQRZP7FZOVJQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.238439144 (601h15m56.109522189s) [noise=true , address=145.79.227.30:4790 , piece_id=5VRI4FYK7J54TCOY7DX6TCQ3634N3TATGDZUZDWRBBLMXCGRVR3Q, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.239316585 (601h15m56.11039962s) [noise=true , address=145.79.227.44:4212 , piece_id=ZV3XMTVGJR2D5X4EEGGDTGGGXSQI4QNCJZCI3GPR6RTD7JEQWLGA, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.242138468 (601h15m56.113221503s) [noise=true , address=45.91.116.11:4131 , piece_id=YAFIDLBM762GWTNKDW62FMNLE4YHCNYW2WU3R4AMO5NELMVOHHHQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.243965547 (601h15m56.115048582s) [noise=true , address=77.90.31.18:4829 , piece_id=MVRLWV5URMPV57R52J426SXY5ZWQGHOTL42GO2GPEZ4LA7B2JGCQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.250805150 (601h15m56.121888185s) [noise=true , address=5.231.73.4:4033 , piece_id=CCH6AH47YBQPGPF4MPZU2NPT7HPF4D3KT3ZVDITRYF7UW5JUKWDQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.254973396 (601h15m56.126056431s) [noise=true , address=145.79.227.49:4341 , piece_id=6GKQQIBOECLDEEDZNO5AYKX7XMXZYUA6WNGKD3FQSF2Q4O6A4MQA, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:57.691193696 (601h15m57.562276731s) [upload=526987]: Next returned nil segment
Dec 30 16:44:57.691206460 (601h15m57.562289495s) [upload=526987]: Waiting for error group managing segments...
Dec 30 16:44:58.085111563 (601h15m57.956194598s) [noise=true , address=64.49.13.137:4493 , piece_id=QZSPIHMAGKSHSAJYJ6BR26EMTCWFAEPOPC4O65YN6SL6222H4HWQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.086614434 (601h15m57.957697479s) [noise=true , address=64.49.13.27:4965 , piece_id=AQTJD7BXKPMIAGEOGJCY5Y64IVU2SHCZEFJL2NLQAHVX5HR33N7Q, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.119421281 (601h15m57.990504316s) [noise=true , address=145.79.227.8:4344 , piece_id=6L5DD7CQJ5JZ7IPFX2IGHNEP7M6BZU2DA5Z5GEOKGPAMVZUXQHOQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.125317667 (601h15m57.996400702s) [noise=true , address=145.79.227.31:4808 , piece_id=MW4UFSJ4XFLXK7ZLWZFWTP2N4QQUS24WP3O44DJWT7AK2WCWAT3Q, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.262371457 (601h15m58.133454502s) [noise=true , address=64.49.13.26:4947 , piece_id=BHVQNSBU3LP35Z5WQYTRPJZNNWTDZKWYK2CMRLHSOGBJ4IXGZ5SA, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.284980216 (601h15m58.156063251s) [noise=true , address=77.90.31.19:4853 , piece_id=MS4HNXUQIITLBMEHZDPJMEH7FSLMBBGKJSIP5BY2BDUF3IE2NKIQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.387549123 (601h15m58.258632158s) [noise=true , address=77.90.31.18:4829 , piece_id=MVRLWV5URMPV57R52J426SXY5ZWQGHOTL42GO2GPEZ4LA7B2JGCQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.394280961 (601h15m58.265364006s) [noise=true , address=5.231.73.4:4033 , piece_id=CCH6AH47YBQPGPF4MPZU2NPT7HPF4D3KT3ZVDITRYF7UW5JUKWDQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.510328564 (601h15m58.381411609s) [noise=true , address=45.91.116.8:4071 , piece_id=3PUQCDJAXSQGEM47BTD2B5HWWNK3PUS6AEVQ63BEC2Q2CKCB4PXQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.545484694 (601h15m58.416567739s) [noise=true , address=45.91.116.11:4131 , piece_id=YAFIDLBM762GWTNKDW62FMNLE4YHCNYW2WU3R4AMO5NELMVOHHHQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.710409562 (601h15m58.581492607s) [noise=true , address=77.90.31.21:4899 , piece_id=DCYPDKTC4S5LG3XEBQWZT3HGFJ63T3UC3PFUDLF7OOJHNMYW4KRQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.981893937 (601h15m58.852976982s) [noise=true , address=145.79.227.44:4212 , piece_id=ZV3XMTVGJR2D5X4EEGGDTGGGXSQI4QNCJZCI3GPR6RTD7JEQWLGA, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.995663946 (601h15m58.866746991s) [noise=true , address=145.79.227.49:4341 , piece_id=6GKQQIBOECLDEEDZNO5AYKX7XMXZYUA6WNGKD3FQSF2Q4O6A4MQA, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:59.009469091 (601h15m58.880552136s) [noise=true , address=145.79.227.40:4064 , piece_id=SEYSDBZHX6SHMFZBGY3XDIYEOTWFCNGG2KNNTCIXYQRZP7FZOVJQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:59.009489950 (601h15m58.880572986s) [seg_pos={0 0}, upload=526987]: Segment reached optimal threshold of 14 pieces.
Dec 30 16:44:59.009662329 (601h15m58.880745374s) [noise=true , address=145.79.227.34:4923 , piece_id=OB3KQEHYXE4BL73EDDQDN2LSVYJYYQQFUSFJ655BFJISLBGOH4XQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VXAxRGiJQor24W7DYSeUQUu1QF7EcX6BQYgfUheACRs2quFFu): protocol: expected piece hash; context canceled; EOF
Dec 30 16:44:59.009668390 (601h15m58.880751425s) [noise=true , address=145.79.227.34:4923 , piece_id=OB3KQEHYXE4BL73EDDQDN2LSVYJYYQQFUSFJ655BFJISLBGOH4XQ, seg_pos={0 0}, upload=526987]: Long tail cancellation due to optimal threshold
Dec 30 16:44:59.009857671 (601h15m58.880940706s) [noise=true , address=145.79.227.33:4903 , piece_id=THSFTSO2Y4QVWIGNUOU4AKFGTSQWQ6FLNIIDISIVNRF7XZZWWPCQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:59.009921232 (601h15m58.881004267s) [noise=true , address=145.79.227.30:4790 , piece_id=5VRI4FYK7J54TCOY7DX6TCQ3634N3TATGDZUZDWRBBLMXCGRVR3Q, seg_pos={0 0}, upload=526987]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kuSMof8EbvsaMCdVxKWjgTT5TfLJnTAeHUf8DKNC3UGQn5q7T): protocol: expected piece hash; context canceled; EOF
Dec 30 16:44:59.009927314 (601h15m58.881010359s) [noise=true , address=145.79.227.30:4790 , piece_id=5VRI4FYK7J54TCOY7DX6TCQ3634N3TATGDZUZDWRBBLMXCGRVR3Q, seg_pos={0 0}, upload=526987]: Long tail cancellation due to optimal threshold
Dec 30 16:44:59.010092589 (601h15m58.881175624s) [noise=true , address=77.90.31.11:4567 , piece_id=NMCHY46TZ6W6EPVLVVCGQXHFERHBLSFB3J55AD32KQSKOVXABO7A, seg_pos={0 0}, upload=526987]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HJVmEuxjrwWCcdwxWYFC5fTULds1kiRH82DiRWipWhRT1Qwdk): protocol: expected piece hash; context canceled; EOF
Dec 30 16:44:59.010101946 (601h15m58.881184981s) [noise=true , address=77.90.31.11:4567 , piece_id=NMCHY46TZ6W6EPVLVVCGQXHFERHBLSFB3J55AD32KQSKOVXABO7A, seg_pos={0 0}, upload=526987]: Long tail cancellation due to optimal threshold
Dec 30 16:44:59.010125410 (601h15m58.881208445s) [seg_pos={0 0}, upload=526987]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil>
Dec 30 16:44:59.010204091 (601h15m58.881287136s) [upload=526987]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 16:44:59.010211274 (601h15m58.881294339s) [upload=526987]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 16:44:59.309660956 (601h15m59.180744001s) [upload=526987]: Done uploading segments...
Dec 30 16:44:59.309679662 (601h15m59.180762697s) [upload=526987]: Upload finished. err: <nil>
Duration: 45.803806269s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
19 out of 60 is failed (threshold=6) [..XXX.X.X.X...XXX..X....XXX..X...XX......X.XX...............] duration:
4.542548893s trace:
https://jaeger.dev.storj.io/trace/2400bb34adb4564c
FAIL (2025-12-30 17:01:54 UTC)
orbiter warning: context deadline exceeded
Duration: 4.5427254s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
23 out of 60 is failed (threshold=3) [..XXXX........X..X....X.XXX..XX.X.X..XXX.XX..X.X.......X.X..] duration:
6.302061973s trace:
https://jaeger.dev.storj.io/trace/5856e1e0aefa8ae4
FAIL (2025-12-30 17:01:47 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
0 16:59:46.957563535 (601h30m46.82864657s) [upload=527201]: Starting upload
Dec 30 16:59:46.957622036 (601h30m46.828705071s) [upload=527201]: Done starting upload
Dec 30 16:59:46.957827287 (601h30m46.828910332s) [upload=527201]: Uploading segments...
Dec 30 16:59:46.957867053 (601h30m46.828950098s) [upload=527201]: Got next segment. Inline: false
Dec 30 16:59:46.957914133 (601h30m46.828997178s) [upload=527201]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 16:59:46.957923240 (601h30m46.829006305s) [upload=527201]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 16:59:47.211654686 (601h30m47.082737721s) [seg_pos={0 0}, upload=527201]: Begin upload segment...
Dec 30 16:59:47.211698439 (601h30m47.082781474s) [seg_pos={0 0}, upload=527201]: Done begin upload segment.
Dec 30 16:59:47.211723978 (601h30m47.082807013s) [noise=true , address=45.91.116.6:4976 , piece_id=X6C732MSJMQNWENN6RLVD62DHRDN3PJALVUH37L5NGBVWVDZVYUQ, seg_pos={0 0}, upload=527201]: Uploading piece...
Dec 30 16:59:47.211781457 (601h30m47.082864492s) [noise=true , address=45.91.116.8:4046 , piece_id=WKQBEPCIFJKI6D733IDPXL3UM6NGOKEVS6XM5FIWSJCPZKNQZ62A, seg_pos={0 0}, upload=527201]: Uploading piece...
Dec 30 16:59:47.212271530 (601h30m47.083354575s) [noise=true , address=145.79.227.48:4302 , piece_id=DGZIHINKA2M3W7L4MVS7ITGI6W2UBSOMZGKZTXXXMD2HVY3HLFOA, seg_pos={0 0}, upload=527201]: Uploading piece...
Dec 30 16:59:47.212602721 (601h30m47.083685766s) [noise=true , address=145.79.227.47:4267 , piece_id=UMVLUUZQXRFNQL7UJLNSVWDXXBIMQNQYEM2EDBW4RYL7XWVNZMVA, seg_pos={0 0}, upload=527201]: Uploading piece...
Dec 30 16:59:47.212995078 (601h30m47.084078113s) [noise=true , address=145.79.227.50:4373 , piece_id=MQF3YUF7U5BK5ZJGSHRCDHWNB4ZMIBWYLQA2XNQHLMUR63L2BBQA, seg_pos={0 0}, upload=527201]: Uploading piece...
Dec 30 16:59:47.213320246 (601h30m47.084403292s) [noise=true , address=145.79.227.18:4007 , piece_id=IXTWS7425QTXR65NJANPFYJOKLP4KCNTSY6JPQWXV2STG7VLNPTQ, seg_pos={0 0}, upload=527201]: Uploading piece...
Dec 30 16:59:47.213639966 (601h30m47.084723001s) [noise=true , address=145.79.227.23:4545 , piece_id=ZUTQSU7R3CIGTTOEK7YETJZDFCBLZUUCMAUUELTYQOHRGMUR5U7Q, seg_pos={0 0}, upload=527201]: Uploading piece...
Dec 30 16:59:47.213961107 (601h30m47.085044142s) [noise=true , address=145.79.227.45:4230 , piece_id=4IXXDMMMJ5Q3KUKH26EQWS5DXHNIFC3PI33QFPYJB6677KNTT5OQ, seg_pos={0 0}, upload=527201]: Uploading piece...
Dec 30 16:59:47.214275286 (601h30m47.085358321s) [noise=true , address=77.90.31.18:4805 , piece_id=SKRCNADR3PEBW6IZS4UK42PORHUNLSWF3YCRNT7YT5QVDAYATWEQ, seg_pos={0 0}, upload=527201]: Uploading piece...
Dec 30 16:59:47.215520627 (601h30m47.086603662s) [noise=true , address=145.79.227.40:4049 , piece_id=UIBS7DURG2M5ARTDGHIOB66GL7GJFH2VDNFHTOHKFCQVTVDTHDVQ, seg_pos={0 0}, upload=527201]: Uploading piece...
Dec 30 16:59:47.216493460 (601h30m47.087576495s) [noise=true , address=145.79.227.5:4222 , piece_id=LA4M42KLBEDPYALFRQHFTE2C6VQ5R6LW6QBZARO6U2ZDRAJPLUEA, seg_pos={0 0}, upload=527201]: Uploading piece...
Dec 30 16:59:47.217694005 (601h30m47.08877704s) [noise=true , address=45.91.116.11:4118 , piece_id=2BJOAC3WIGLSXPVXGDN4ORGVENIRZ6ZNFI4GT2SROQPETNQNXHQQ, seg_pos={0 0}, upload=527201]: Uploading piece...
Dec 30 16:59:47.218989001 (601h30m47.090072036s) [noise=true , address=145.79.227.42:4123 , piece_id=NQBH6QQPUNNDA3CCSAOITPVQ3SAO77E42W2FDBPF4GRTYD6VSWFQ, seg_pos={0 0}, upload=527201]: Uploading piece...
Dec 30 16:59:47.220326387 (601h30m47.091409432s) [noise=true , address=145.79.227.31:4797 , piece_id=M4MB7O3EA3NDFGKYBKZOPLM7JDU4IXP6HG4XHL72O4GJKSQFTKAQ, seg_pos={0 0}, upload=527201]: Uploading piece...
Dec 30 16:59:47.221645088 (601h30m47.092728123s) [noise=true , address=145.79.227.6:4259 , piece_id=RM4WPRLPBQJRA2B2AR6CDZXO7QBFASUXGCP2NDOZI6LWM4HTHB2A, seg_pos={0 0}, upload=527201]: Uploading piece...
Dec 30 16:59:47.222831628 (601h30m47.093914663s) [noise=true , address=77.90.31.19:4841 , piece_id=5CVPDBWZZMTF4RU7YLDDI4HJ57H7RMGILPX7PIMFPTNNC4MFKQEQ, seg_pos={0 0}, upload=527201]: Uploading piece...
Dec 30 16:59:48.903207088 (601h30m48.774290123s) [upload=527201]: Next returned nil segment
Dec 30 16:59:48.903238007 (601h30m48.774321042s) [upload=527201]: Waiting for error group managing segments...
Dec 30 16:59:49.187976684 (601h30m49.059059729s) [noise=true , address=145.79.227.45:4230 , piece_id=4IXXDMMMJ5Q3KUKH26EQWS5DXHNIFC3PI33QFPYJB6677KNTT5OQ, seg_pos={0 0}, upload=527201]: Done uploading piece. err: <nil>
Dec 30 16:59:49.198069342 (601h30m49.069152377s) [noise=true , address=145.79.227.50:4373 , piece_id=MQF3YUF7U5BK5ZJGSHRCDHWNB4ZMIBWYLQA2XNQHLMUR63L2BBQA, seg_pos={0 0}, upload=527201]: Done uploading piece. err: <nil>
Dec 30 16:59:49.202160432 (601h30m49.073243467s) [noise=true , address=145.79.227.48:4302 , piece_id=DGZIHINKA2M3W7L4MVS7ITGI6W2UBSOMZGKZTXXXMD2HVY3HLFOA, seg_pos={0 0}, upload=527201]: Done uploading piece. err: <nil>
Dec 30 16:59:49.213483222 (601h30m49.084566257s) [noise=true , address=145.79.227.42:4123 , piece_id=NQBH6QQPUNNDA3CCSAOITPVQ3SAO77E42W2FDBPF4GRTYD6VSWFQ, seg_pos={0 0}, upload=527201]: Done uploading piece. err: <nil>
Dec 30 16:59:49.229077405 (601h30m49.10016045s) [noise=true , address=145.79.227.31:4797 , piece_id=M4MB7O3EA3NDFGKYBKZOPLM7JDU4IXP6HG4XHL72O4GJKSQFTKAQ, seg_pos={0 0}, upload=527201]: Done uploading piece. err: <nil>
Dec 30 16:59:49.232909141 (601h30m49.103992176s) [noise=true , address=145.79.227.47:4267 , piece_id=UMVLUUZQXRFNQL7UJLNSVWDXXBIMQNQYEM2EDBW4RYL7XWVNZMVA, seg_pos={0 0}, upload=527201]: Done uploading piece. err: <nil>
Dec 30 16:59:49.233336004 (601h30m49.104419039s) [noise=true , address=145.79.227.18:4007 , piece_id=IXTWS7425QTXR65NJANPFYJOKLP4KCNTSY6JPQWXV2STG7VLNPTQ, seg_pos={0 0}, upload=527201]: Done uploading piece. err: <nil>
Dec 30 16:59:49.291279996 (601h30m49.162363041s) [noise=true , address=145.79.227.5:4222 , piece_id=LA4M42KLBEDPYALFRQHFTE2C6VQ5R6LW6QBZARO6U2ZDRAJPLUEA, seg_pos={0 0}, upload=527201]: Done uploading piece. err: <nil>
Dec 30 16:59:49.301745013 (601h30m49.172828048s) [noise=true , address=145.79.227.6:4259 , piece_id=RM4WPRLPBQJRA2B2AR6CDZXO7QBFASUXGCP2NDOZI6LWM4HTHB2A, seg_pos={0 0}, upload=527201]: Done uploading piece. err: <nil>
Dec 30 16:59:49.373505102 (601h30m49.244588137s) [noise=true , address=145.79.227.23:4545 , piece_id=ZUTQSU7R3CIGTTOEK7YETJZDFCBLZUUCMAUUELTYQOHRGMUR5U7Q, seg_pos={0 0}, upload=527201]: Done uploading piece. err: <nil>
Dec 30 16:59:49.537360624 (601h30m49.408443659s) [noise=true , address=45.91.116.8:4046 , piece_id=WKQBEPCIFJKI6D733IDPXL3UM6NGOKEVS6XM5FIWSJCPZKNQZ62A, seg_pos={0 0}, upload=527201]: Done uploading piece. err: <nil>
Dec 30 16:59:49.634981796 (601h30m49.506064831s) [noise=true , address=45.91.116.6:4976 , piece_id=X6C732MSJMQNWENN6RLVD62DHRDN3PJALVUH37L5NGBVWVDZVYUQ, seg_pos={0 0}, upload=527201]: Done uploading piece. err: <nil>
Dec 30 16:59:49.791539009 (601h30m49.662622044s) [noise=true , address=77.90.31.19:4841 , piece_id=5CVPDBWZZMTF4RU7YLDDI4HJ57H7RMGILPX7PIMFPTNNC4MFKQEQ, seg_pos={0 0}, upload=527201]: Done uploading piece. err: <nil>
Dec 30 16:59:49.791566211 (601h30m49.662649257s) [seg_pos={0 0}, upload=527201]: Segment reached optimal threshold of 13 pieces.
Dec 30 16:59:49.791768326 (601h30m49.662851371s) [noise=true , address=145.79.227.40:4049 , piece_id=UIBS7DURG2M5ARTDGHIOB66GL7GJFH2VDNFHTOHKFCQVTVDTHDVQ, seg_pos={0 0}, upload=527201]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF
Dec 30 16:59:49.791776422 (601h30m49.662859457s) [noise=true , address=145.79.227.40:4049 , piece_id=UIBS7DURG2M5ARTDGHIOB66GL7GJFH2VDNFHTOHKFCQVTVDTHDVQ, seg_pos={0 0}, upload=527201]: Long tail cancellation due to optimal threshold
Dec 30 16:59:49.791794476 (601h30m49.662877511s) [noise=true , address=45.91.116.11:4118 , piece_id=2BJOAC3WIGLSXPVXGDN4ORGVENIRZ6ZNFI4GT2SROQPETNQNXHQQ, seg_pos={0 0}, upload=527201]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF
Dec 30 16:59:49.791802100 (601h30m49.662885135s) [noise=true , address=45.91.116.11:4118 , piece_id=2BJOAC3WIGLSXPVXGDN4ORGVENIRZ6ZNFI4GT2SROQPETNQNXHQQ, seg_pos={0 0}, upload=527201]: Long tail cancellation due to optimal threshold
Dec 30 16:59:49.792001189 (601h30m49.663084224s) [noise=true , address=77.90.31.18:4805 , piece_id=SKRCNADR3PEBW6IZS4UK42PORHUNLSWF3YCRNT7YT5QVDAYATWEQ, seg_pos={0 0}, upload=527201]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:53050->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:53050->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53050->77.90.31.18:4805: use of closed network connection
Dec 30 16:59:49.792007641 (601h30m49.663090686s) [noise=true , address=77.90.31.18:4805 , piece_id=SKRCNADR3PEBW6IZS4UK42PORHUNLSWF3YCRNT7YT5QVDAYATWEQ, seg_pos={0 0}, upload=527201]: Long tail cancellation due to optimal threshold
Dec 30 16:59:49.792022360 (601h30m49.663105395s) [seg_pos={0 0}, upload=527201]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 16:59:49.792089899 (601h30m49.663172934s) [upload=527201]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 16:59:49.792096441 (601h30m49.663179476s) [upload=527201]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 16:59:50.076398858 (601h30m49.947481913s) [upload=527201]: Done uploading segments...
Dec 30 16:59:50.076445606 (601h30m49.947528651s) [upload=527201]: Upload finished. err: <nil>
Duration: 6.302086741s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [................X.............X.......X.....................] duration:
32.064694384s trace:
https://jaeger.dev.storj.io/trace/3f4379d3637c8e4b
FAIL (2025-12-30 17:01:35 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1WBuBdGxdPXi6VsNf4FJpn3LpWoQztM7CV6YjAW1mUDVjAUjn2, Piece ID: MS4HNXUQIITLBMEHZDPJMEH7FSLMBBGKJSIP5BY2BDUF3IE2NKIQ) context deadline exceeded; piecestore: (Node ID: 1soiTQsiMNe5BkNHwwAGykmtP9LtLxncqE7Tv728ak8pjFeshg, Piece ID: BHVQNSBU3LP35Z5WQYTRPJZNNWTDZKWYK2CMRLHSOGBJ4IXGZ5SA) context deadline exceeded; piecestore: (Node ID: 1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, Piece ID: MVRLWV5URMPV57R52J426SXY5ZWQGHOTL42GO2GPEZ4LA7B2JGCQ) context deadline exceeded
======LOGS======
5m56.099625936s) [seg_pos={0 0}, upload=526987]: Begin upload segment...
Dec 30 16:44:56.228605921 (601h15m56.099688967s) [seg_pos={0 0}, upload=526987]: Done begin upload segment.
Dec 30 16:44:56.228736641 (601h15m56.099819686s) [noise=true , address=145.79.227.34:4923 , piece_id=OB3KQEHYXE4BL73EDDQDN2LSVYJYYQQFUSFJ655BFJISLBGOH4XQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.228753061 (601h15m56.099836117s) [noise=true , address=64.49.13.26:4947 , piece_id=BHVQNSBU3LP35Z5WQYTRPJZNNWTDZKWYK2CMRLHSOGBJ4IXGZ5SA, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.229701116 (601h15m56.100784151s) [noise=true , address=45.91.116.8:4071 , piece_id=3PUQCDJAXSQGEM47BTD2B5HWWNK3PUS6AEVQ63BEC2Q2CKCB4PXQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.230399837 (601h15m56.101482882s) [noise=true , address=77.90.31.21:4899 , piece_id=DCYPDKTC4S5LG3XEBQWZT3HGFJ63T3UC3PFUDLF7OOJHNMYW4KRQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.230834926 (601h15m56.101917971s) [noise=true , address=77.90.31.19:4853 , piece_id=MS4HNXUQIITLBMEHZDPJMEH7FSLMBBGKJSIP5BY2BDUF3IE2NKIQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.232511007 (601h15m56.103594112s) [noise=true , address=145.79.227.8:4344 , piece_id=6L5DD7CQJ5JZ7IPFX2IGHNEP7M6BZU2DA5Z5GEOKGPAMVZUXQHOQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.233047428 (601h15m56.104130473s) [noise=true , address=145.79.227.31:4808 , piece_id=MW4UFSJ4XFLXK7ZLWZFWTP2N4QQUS24WP3O44DJWT7AK2WCWAT3Q, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.233557500 (601h15m56.104640545s) [noise=true , address=145.79.227.33:4903 , piece_id=THSFTSO2Y4QVWIGNUOU4AKFGTSQWQ6FLNIIDISIVNRF7XZZWWPCQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.234065937 (601h15m56.105148983s) [noise=true , address=64.49.13.137:4493 , piece_id=QZSPIHMAGKSHSAJYJ6BR26EMTCWFAEPOPC4O65YN6SL6222H4HWQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.234513860 (601h15m56.105596896s) [noise=true , address=64.49.13.27:4965 , piece_id=AQTJD7BXKPMIAGEOGJCY5Y64IVU2SHCZEFJL2NLQAHVX5HR33N7Q, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.236125519 (601h15m56.107208554s) [noise=true , address=77.90.31.11:4567 , piece_id=NMCHY46TZ6W6EPVLVVCGQXHFERHBLSFB3J55AD32KQSKOVXABO7A, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.237179756 (601h15m56.108262791s) [noise=true , address=145.79.227.40:4064 , piece_id=SEYSDBZHX6SHMFZBGY3XDIYEOTWFCNGG2KNNTCIXYQRZP7FZOVJQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.238439144 (601h15m56.109522189s) [noise=true , address=145.79.227.30:4790 , piece_id=5VRI4FYK7J54TCOY7DX6TCQ3634N3TATGDZUZDWRBBLMXCGRVR3Q, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.239316585 (601h15m56.11039962s) [noise=true , address=145.79.227.44:4212 , piece_id=ZV3XMTVGJR2D5X4EEGGDTGGGXSQI4QNCJZCI3GPR6RTD7JEQWLGA, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.242138468 (601h15m56.113221503s) [noise=true , address=45.91.116.11:4131 , piece_id=YAFIDLBM762GWTNKDW62FMNLE4YHCNYW2WU3R4AMO5NELMVOHHHQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.243965547 (601h15m56.115048582s) [noise=true , address=77.90.31.18:4829 , piece_id=MVRLWV5URMPV57R52J426SXY5ZWQGHOTL42GO2GPEZ4LA7B2JGCQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.250805150 (601h15m56.121888185s) [noise=true , address=5.231.73.4:4033 , piece_id=CCH6AH47YBQPGPF4MPZU2NPT7HPF4D3KT3ZVDITRYF7UW5JUKWDQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.254973396 (601h15m56.126056431s) [noise=true , address=145.79.227.49:4341 , piece_id=6GKQQIBOECLDEEDZNO5AYKX7XMXZYUA6WNGKD3FQSF2Q4O6A4MQA, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:57.691193696 (601h15m57.562276731s) [upload=526987]: Next returned nil segment
Dec 30 16:44:57.691206460 (601h15m57.562289495s) [upload=526987]: Waiting for error group managing segments...
Dec 30 16:44:58.085111563 (601h15m57.956194598s) [noise=true , address=64.49.13.137:4493 , piece_id=QZSPIHMAGKSHSAJYJ6BR26EMTCWFAEPOPC4O65YN6SL6222H4HWQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.086614434 (601h15m57.957697479s) [noise=true , address=64.49.13.27:4965 , piece_id=AQTJD7BXKPMIAGEOGJCY5Y64IVU2SHCZEFJL2NLQAHVX5HR33N7Q, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.119421281 (601h15m57.990504316s) [noise=true , address=145.79.227.8:4344 , piece_id=6L5DD7CQJ5JZ7IPFX2IGHNEP7M6BZU2DA5Z5GEOKGPAMVZUXQHOQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.125317667 (601h15m57.996400702s) [noise=true , address=145.79.227.31:4808 , piece_id=MW4UFSJ4XFLXK7ZLWZFWTP2N4QQUS24WP3O44DJWT7AK2WCWAT3Q, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.262371457 (601h15m58.133454502s) [noise=true , address=64.49.13.26:4947 , piece_id=BHVQNSBU3LP35Z5WQYTRPJZNNWTDZKWYK2CMRLHSOGBJ4IXGZ5SA, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.284980216 (601h15m58.156063251s) [noise=true , address=77.90.31.19:4853 , piece_id=MS4HNXUQIITLBMEHZDPJMEH7FSLMBBGKJSIP5BY2BDUF3IE2NKIQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.387549123 (601h15m58.258632158s) [noise=true , address=77.90.31.18:4829 , piece_id=MVRLWV5URMPV57R52J426SXY5ZWQGHOTL42GO2GPEZ4LA7B2JGCQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.394280961 (601h15m58.265364006s) [noise=true , address=5.231.73.4:4033 , piece_id=CCH6AH47YBQPGPF4MPZU2NPT7HPF4D3KT3ZVDITRYF7UW5JUKWDQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.510328564 (601h15m58.381411609s) [noise=true , address=45.91.116.8:4071 , piece_id=3PUQCDJAXSQGEM47BTD2B5HWWNK3PUS6AEVQ63BEC2Q2CKCB4PXQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.545484694 (601h15m58.416567739s) [noise=true , address=45.91.116.11:4131 , piece_id=YAFIDLBM762GWTNKDW62FMNLE4YHCNYW2WU3R4AMO5NELMVOHHHQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.710409562 (601h15m58.581492607s) [noise=true , address=77.90.31.21:4899 , piece_id=DCYPDKTC4S5LG3XEBQWZT3HGFJ63T3UC3PFUDLF7OOJHNMYW4KRQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.981893937 (601h15m58.852976982s) [noise=true , address=145.79.227.44:4212 , piece_id=ZV3XMTVGJR2D5X4EEGGDTGGGXSQI4QNCJZCI3GPR6RTD7JEQWLGA, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.995663946 (601h15m58.866746991s) [noise=true , address=145.79.227.49:4341 , piece_id=6GKQQIBOECLDEEDZNO5AYKX7XMXZYUA6WNGKD3FQSF2Q4O6A4MQA, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:59.009469091 (601h15m58.880552136s) [noise=true , address=145.79.227.40:4064 , piece_id=SEYSDBZHX6SHMFZBGY3XDIYEOTWFCNGG2KNNTCIXYQRZP7FZOVJQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:59.009489950 (601h15m58.880572986s) [seg_pos={0 0}, upload=526987]: Segment reached optimal threshold of 14 pieces.
Dec 30 16:44:59.009662329 (601h15m58.880745374s) [noise=true , address=145.79.227.34:4923 , piece_id=OB3KQEHYXE4BL73EDDQDN2LSVYJYYQQFUSFJ655BFJISLBGOH4XQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VXAxRGiJQor24W7DYSeUQUu1QF7EcX6BQYgfUheACRs2quFFu): protocol: expected piece hash; context canceled; EOF
Dec 30 16:44:59.009668390 (601h15m58.880751425s) [noise=true , address=145.79.227.34:4923 , piece_id=OB3KQEHYXE4BL73EDDQDN2LSVYJYYQQFUSFJ655BFJISLBGOH4XQ, seg_pos={0 0}, upload=526987]: Long tail cancellation due to optimal threshold
Dec 30 16:44:59.009857671 (601h15m58.880940706s) [noise=true , address=145.79.227.33:4903 , piece_id=THSFTSO2Y4QVWIGNUOU4AKFGTSQWQ6FLNIIDISIVNRF7XZZWWPCQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:59.009921232 (601h15m58.881004267s) [noise=true , address=145.79.227.30:4790 , piece_id=5VRI4FYK7J54TCOY7DX6TCQ3634N3TATGDZUZDWRBBLMXCGRVR3Q, seg_pos={0 0}, upload=526987]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kuSMof8EbvsaMCdVxKWjgTT5TfLJnTAeHUf8DKNC3UGQn5q7T): protocol: expected piece hash; context canceled; EOF
Dec 30 16:44:59.009927314 (601h15m58.881010359s) [noise=true , address=145.79.227.30:4790 , piece_id=5VRI4FYK7J54TCOY7DX6TCQ3634N3TATGDZUZDWRBBLMXCGRVR3Q, seg_pos={0 0}, upload=526987]: Long tail cancellation due to optimal threshold
Dec 30 16:44:59.010092589 (601h15m58.881175624s) [noise=true , address=77.90.31.11:4567 , piece_id=NMCHY46TZ6W6EPVLVVCGQXHFERHBLSFB3J55AD32KQSKOVXABO7A, seg_pos={0 0}, upload=526987]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HJVmEuxjrwWCcdwxWYFC5fTULds1kiRH82DiRWipWhRT1Qwdk): protocol: expected piece hash; context canceled; EOF
Dec 30 16:44:59.010101946 (601h15m58.881184981s) [noise=true , address=77.90.31.11:4567 , piece_id=NMCHY46TZ6W6EPVLVVCGQXHFERHBLSFB3J55AD32KQSKOVXABO7A, seg_pos={0 0}, upload=526987]: Long tail cancellation due to optimal threshold
Dec 30 16:44:59.010125410 (601h15m58.881208445s) [seg_pos={0 0}, upload=526987]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil>
Dec 30 16:44:59.010204091 (601h15m58.881287136s) [upload=526987]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 16:44:59.010211274 (601h15m58.881294339s) [upload=526987]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 16:44:59.309660956 (601h15m59.180744001s) [upload=526987]: Done uploading segments...
Dec 30 16:44:59.309679662 (601h15m59.180762697s) [upload=526987]: Upload finished. err: <nil>
Duration: 32.064726144s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
19 out of 60 is failed (threshold=6) [...XXX.X.X.X...XXX..X....XXX..X...XX......X.XX..............] duration:
4.398025172s trace:
https://jaeger.dev.storj.io/trace/2400bb34adb4564c
FAIL (2025-12-30 17:01:02 UTC)
orbiter warning: context deadline exceeded
Duration: 4.398069056s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [.................X.............X.......X....................] duration:
19.236382248s trace:
https://jaeger.dev.storj.io/trace/3f4379d3637c8e4b
FAIL (2025-12-30 17:00:55 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1WBuBdGxdPXi6VsNf4FJpn3LpWoQztM7CV6YjAW1mUDVjAUjn2, Piece ID: MS4HNXUQIITLBMEHZDPJMEH7FSLMBBGKJSIP5BY2BDUF3IE2NKIQ) context deadline exceeded; piecestore: (Node ID: 1soiTQsiMNe5BkNHwwAGykmtP9LtLxncqE7Tv728ak8pjFeshg, Piece ID: BHVQNSBU3LP35Z5WQYTRPJZNNWTDZKWYK2CMRLHSOGBJ4IXGZ5SA) context deadline exceeded; piecestore: (Node ID: 1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, Piece ID: MVRLWV5URMPV57R52J426SXY5ZWQGHOTL42GO2GPEZ4LA7B2JGCQ) context deadline exceeded
======LOGS======
5m56.099625936s) [seg_pos={0 0}, upload=526987]: Begin upload segment...
Dec 30 16:44:56.228605921 (601h15m56.099688967s) [seg_pos={0 0}, upload=526987]: Done begin upload segment.
Dec 30 16:44:56.228736641 (601h15m56.099819686s) [noise=true , address=145.79.227.34:4923 , piece_id=OB3KQEHYXE4BL73EDDQDN2LSVYJYYQQFUSFJ655BFJISLBGOH4XQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.228753061 (601h15m56.099836117s) [noise=true , address=64.49.13.26:4947 , piece_id=BHVQNSBU3LP35Z5WQYTRPJZNNWTDZKWYK2CMRLHSOGBJ4IXGZ5SA, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.229701116 (601h15m56.100784151s) [noise=true , address=45.91.116.8:4071 , piece_id=3PUQCDJAXSQGEM47BTD2B5HWWNK3PUS6AEVQ63BEC2Q2CKCB4PXQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.230399837 (601h15m56.101482882s) [noise=true , address=77.90.31.21:4899 , piece_id=DCYPDKTC4S5LG3XEBQWZT3HGFJ63T3UC3PFUDLF7OOJHNMYW4KRQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.230834926 (601h15m56.101917971s) [noise=true , address=77.90.31.19:4853 , piece_id=MS4HNXUQIITLBMEHZDPJMEH7FSLMBBGKJSIP5BY2BDUF3IE2NKIQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.232511007 (601h15m56.103594112s) [noise=true , address=145.79.227.8:4344 , piece_id=6L5DD7CQJ5JZ7IPFX2IGHNEP7M6BZU2DA5Z5GEOKGPAMVZUXQHOQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.233047428 (601h15m56.104130473s) [noise=true , address=145.79.227.31:4808 , piece_id=MW4UFSJ4XFLXK7ZLWZFWTP2N4QQUS24WP3O44DJWT7AK2WCWAT3Q, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.233557500 (601h15m56.104640545s) [noise=true , address=145.79.227.33:4903 , piece_id=THSFTSO2Y4QVWIGNUOU4AKFGTSQWQ6FLNIIDISIVNRF7XZZWWPCQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.234065937 (601h15m56.105148983s) [noise=true , address=64.49.13.137:4493 , piece_id=QZSPIHMAGKSHSAJYJ6BR26EMTCWFAEPOPC4O65YN6SL6222H4HWQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.234513860 (601h15m56.105596896s) [noise=true , address=64.49.13.27:4965 , piece_id=AQTJD7BXKPMIAGEOGJCY5Y64IVU2SHCZEFJL2NLQAHVX5HR33N7Q, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.236125519 (601h15m56.107208554s) [noise=true , address=77.90.31.11:4567 , piece_id=NMCHY46TZ6W6EPVLVVCGQXHFERHBLSFB3J55AD32KQSKOVXABO7A, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.237179756 (601h15m56.108262791s) [noise=true , address=145.79.227.40:4064 , piece_id=SEYSDBZHX6SHMFZBGY3XDIYEOTWFCNGG2KNNTCIXYQRZP7FZOVJQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.238439144 (601h15m56.109522189s) [noise=true , address=145.79.227.30:4790 , piece_id=5VRI4FYK7J54TCOY7DX6TCQ3634N3TATGDZUZDWRBBLMXCGRVR3Q, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.239316585 (601h15m56.11039962s) [noise=true , address=145.79.227.44:4212 , piece_id=ZV3XMTVGJR2D5X4EEGGDTGGGXSQI4QNCJZCI3GPR6RTD7JEQWLGA, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.242138468 (601h15m56.113221503s) [noise=true , address=45.91.116.11:4131 , piece_id=YAFIDLBM762GWTNKDW62FMNLE4YHCNYW2WU3R4AMO5NELMVOHHHQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.243965547 (601h15m56.115048582s) [noise=true , address=77.90.31.18:4829 , piece_id=MVRLWV5URMPV57R52J426SXY5ZWQGHOTL42GO2GPEZ4LA7B2JGCQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.250805150 (601h15m56.121888185s) [noise=true , address=5.231.73.4:4033 , piece_id=CCH6AH47YBQPGPF4MPZU2NPT7HPF4D3KT3ZVDITRYF7UW5JUKWDQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.254973396 (601h15m56.126056431s) [noise=true , address=145.79.227.49:4341 , piece_id=6GKQQIBOECLDEEDZNO5AYKX7XMXZYUA6WNGKD3FQSF2Q4O6A4MQA, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:57.691193696 (601h15m57.562276731s) [upload=526987]: Next returned nil segment
Dec 30 16:44:57.691206460 (601h15m57.562289495s) [upload=526987]: Waiting for error group managing segments...
Dec 30 16:44:58.085111563 (601h15m57.956194598s) [noise=true , address=64.49.13.137:4493 , piece_id=QZSPIHMAGKSHSAJYJ6BR26EMTCWFAEPOPC4O65YN6SL6222H4HWQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.086614434 (601h15m57.957697479s) [noise=true , address=64.49.13.27:4965 , piece_id=AQTJD7BXKPMIAGEOGJCY5Y64IVU2SHCZEFJL2NLQAHVX5HR33N7Q, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.119421281 (601h15m57.990504316s) [noise=true , address=145.79.227.8:4344 , piece_id=6L5DD7CQJ5JZ7IPFX2IGHNEP7M6BZU2DA5Z5GEOKGPAMVZUXQHOQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.125317667 (601h15m57.996400702s) [noise=true , address=145.79.227.31:4808 , piece_id=MW4UFSJ4XFLXK7ZLWZFWTP2N4QQUS24WP3O44DJWT7AK2WCWAT3Q, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.262371457 (601h15m58.133454502s) [noise=true , address=64.49.13.26:4947 , piece_id=BHVQNSBU3LP35Z5WQYTRPJZNNWTDZKWYK2CMRLHSOGBJ4IXGZ5SA, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.284980216 (601h15m58.156063251s) [noise=true , address=77.90.31.19:4853 , piece_id=MS4HNXUQIITLBMEHZDPJMEH7FSLMBBGKJSIP5BY2BDUF3IE2NKIQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.387549123 (601h15m58.258632158s) [noise=true , address=77.90.31.18:4829 , piece_id=MVRLWV5URMPV57R52J426SXY5ZWQGHOTL42GO2GPEZ4LA7B2JGCQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.394280961 (601h15m58.265364006s) [noise=true , address=5.231.73.4:4033 , piece_id=CCH6AH47YBQPGPF4MPZU2NPT7HPF4D3KT3ZVDITRYF7UW5JUKWDQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.510328564 (601h15m58.381411609s) [noise=true , address=45.91.116.8:4071 , piece_id=3PUQCDJAXSQGEM47BTD2B5HWWNK3PUS6AEVQ63BEC2Q2CKCB4PXQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.545484694 (601h15m58.416567739s) [noise=true , address=45.91.116.11:4131 , piece_id=YAFIDLBM762GWTNKDW62FMNLE4YHCNYW2WU3R4AMO5NELMVOHHHQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.710409562 (601h15m58.581492607s) [noise=true , address=77.90.31.21:4899 , piece_id=DCYPDKTC4S5LG3XEBQWZT3HGFJ63T3UC3PFUDLF7OOJHNMYW4KRQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.981893937 (601h15m58.852976982s) [noise=true , address=145.79.227.44:4212 , piece_id=ZV3XMTVGJR2D5X4EEGGDTGGGXSQI4QNCJZCI3GPR6RTD7JEQWLGA, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.995663946 (601h15m58.866746991s) [noise=true , address=145.79.227.49:4341 , piece_id=6GKQQIBOECLDEEDZNO5AYKX7XMXZYUA6WNGKD3FQSF2Q4O6A4MQA, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:59.009469091 (601h15m58.880552136s) [noise=true , address=145.79.227.40:4064 , piece_id=SEYSDBZHX6SHMFZBGY3XDIYEOTWFCNGG2KNNTCIXYQRZP7FZOVJQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:59.009489950 (601h15m58.880572986s) [seg_pos={0 0}, upload=526987]: Segment reached optimal threshold of 14 pieces.
Dec 30 16:44:59.009662329 (601h15m58.880745374s) [noise=true , address=145.79.227.34:4923 , piece_id=OB3KQEHYXE4BL73EDDQDN2LSVYJYYQQFUSFJ655BFJISLBGOH4XQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VXAxRGiJQor24W7DYSeUQUu1QF7EcX6BQYgfUheACRs2quFFu): protocol: expected piece hash; context canceled; EOF
Dec 30 16:44:59.009668390 (601h15m58.880751425s) [noise=true , address=145.79.227.34:4923 , piece_id=OB3KQEHYXE4BL73EDDQDN2LSVYJYYQQFUSFJ655BFJISLBGOH4XQ, seg_pos={0 0}, upload=526987]: Long tail cancellation due to optimal threshold
Dec 30 16:44:59.009857671 (601h15m58.880940706s) [noise=true , address=145.79.227.33:4903 , piece_id=THSFTSO2Y4QVWIGNUOU4AKFGTSQWQ6FLNIIDISIVNRF7XZZWWPCQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:59.009921232 (601h15m58.881004267s) [noise=true , address=145.79.227.30:4790 , piece_id=5VRI4FYK7J54TCOY7DX6TCQ3634N3TATGDZUZDWRBBLMXCGRVR3Q, seg_pos={0 0}, upload=526987]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kuSMof8EbvsaMCdVxKWjgTT5TfLJnTAeHUf8DKNC3UGQn5q7T): protocol: expected piece hash; context canceled; EOF
Dec 30 16:44:59.009927314 (601h15m58.881010359s) [noise=true , address=145.79.227.30:4790 , piece_id=5VRI4FYK7J54TCOY7DX6TCQ3634N3TATGDZUZDWRBBLMXCGRVR3Q, seg_pos={0 0}, upload=526987]: Long tail cancellation due to optimal threshold
Dec 30 16:44:59.010092589 (601h15m58.881175624s) [noise=true , address=77.90.31.11:4567 , piece_id=NMCHY46TZ6W6EPVLVVCGQXHFERHBLSFB3J55AD32KQSKOVXABO7A, seg_pos={0 0}, upload=526987]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HJVmEuxjrwWCcdwxWYFC5fTULds1kiRH82DiRWipWhRT1Qwdk): protocol: expected piece hash; context canceled; EOF
Dec 30 16:44:59.010101946 (601h15m58.881184981s) [noise=true , address=77.90.31.11:4567 , piece_id=NMCHY46TZ6W6EPVLVVCGQXHFERHBLSFB3J55AD32KQSKOVXABO7A, seg_pos={0 0}, upload=526987]: Long tail cancellation due to optimal threshold
Dec 30 16:44:59.010125410 (601h15m58.881208445s) [seg_pos={0 0}, upload=526987]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil>
Dec 30 16:44:59.010204091 (601h15m58.881287136s) [upload=526987]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 16:44:59.010211274 (601h15m58.881294339s) [upload=526987]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 16:44:59.309660956 (601h15m59.180744001s) [upload=526987]: Done uploading segments...
Dec 30 16:44:59.309679662 (601h15m59.180762697s) [upload=526987]: Upload finished. err: <nil>
Duration: 19.2363986s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
24 out of 60 is failed (threshold=3) [X..XXXX........X..X....X.XXX..XX.X.X..XXX.XX..X.X.......X.X.] duration:
6.057585596s trace:
https://jaeger.dev.storj.io/trace/5856e1e0aefa8ae4
FAIL (2025-12-30 17:00:49 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
0 16:59:46.957563535 (601h30m46.82864657s) [upload=527201]: Starting upload
Dec 30 16:59:46.957622036 (601h30m46.828705071s) [upload=527201]: Done starting upload
Dec 30 16:59:46.957827287 (601h30m46.828910332s) [upload=527201]: Uploading segments...
Dec 30 16:59:46.957867053 (601h30m46.828950098s) [upload=527201]: Got next segment. Inline: false
Dec 30 16:59:46.957914133 (601h30m46.828997178s) [upload=527201]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 16:59:46.957923240 (601h30m46.829006305s) [upload=527201]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 16:59:47.211654686 (601h30m47.082737721s) [seg_pos={0 0}, upload=527201]: Begin upload segment...
Dec 30 16:59:47.211698439 (601h30m47.082781474s) [seg_pos={0 0}, upload=527201]: Done begin upload segment.
Dec 30 16:59:47.211723978 (601h30m47.082807013s) [noise=true , address=45.91.116.6:4976 , piece_id=X6C732MSJMQNWENN6RLVD62DHRDN3PJALVUH37L5NGBVWVDZVYUQ, seg_pos={0 0}, upload=527201]: Uploading piece...
Dec 30 16:59:47.211781457 (601h30m47.082864492s) [noise=true , address=45.91.116.8:4046 , piece_id=WKQBEPCIFJKI6D733IDPXL3UM6NGOKEVS6XM5FIWSJCPZKNQZ62A, seg_pos={0 0}, upload=527201]: Uploading piece...
Dec 30 16:59:47.212271530 (601h30m47.083354575s) [noise=true , address=145.79.227.48:4302 , piece_id=DGZIHINKA2M3W7L4MVS7ITGI6W2UBSOMZGKZTXXXMD2HVY3HLFOA, seg_pos={0 0}, upload=527201]: Uploading piece...
Dec 30 16:59:47.212602721 (601h30m47.083685766s) [noise=true , address=145.79.227.47:4267 , piece_id=UMVLUUZQXRFNQL7UJLNSVWDXXBIMQNQYEM2EDBW4RYL7XWVNZMVA, seg_pos={0 0}, upload=527201]: Uploading piece...
Dec 30 16:59:47.212995078 (601h30m47.084078113s) [noise=true , address=145.79.227.50:4373 , piece_id=MQF3YUF7U5BK5ZJGSHRCDHWNB4ZMIBWYLQA2XNQHLMUR63L2BBQA, seg_pos={0 0}, upload=527201]: Uploading piece...
Dec 30 16:59:47.213320246 (601h30m47.084403292s) [noise=true , address=145.79.227.18:4007 , piece_id=IXTWS7425QTXR65NJANPFYJOKLP4KCNTSY6JPQWXV2STG7VLNPTQ, seg_pos={0 0}, upload=527201]: Uploading piece...
Dec 30 16:59:47.213639966 (601h30m47.084723001s) [noise=true , address=145.79.227.23:4545 , piece_id=ZUTQSU7R3CIGTTOEK7YETJZDFCBLZUUCMAUUELTYQOHRGMUR5U7Q, seg_pos={0 0}, upload=527201]: Uploading piece...
Dec 30 16:59:47.213961107 (601h30m47.085044142s) [noise=true , address=145.79.227.45:4230 , piece_id=4IXXDMMMJ5Q3KUKH26EQWS5DXHNIFC3PI33QFPYJB6677KNTT5OQ, seg_pos={0 0}, upload=527201]: Uploading piece...
Dec 30 16:59:47.214275286 (601h30m47.085358321s) [noise=true , address=77.90.31.18:4805 , piece_id=SKRCNADR3PEBW6IZS4UK42PORHUNLSWF3YCRNT7YT5QVDAYATWEQ, seg_pos={0 0}, upload=527201]: Uploading piece...
Dec 30 16:59:47.215520627 (601h30m47.086603662s) [noise=true , address=145.79.227.40:4049 , piece_id=UIBS7DURG2M5ARTDGHIOB66GL7GJFH2VDNFHTOHKFCQVTVDTHDVQ, seg_pos={0 0}, upload=527201]: Uploading piece...
Dec 30 16:59:47.216493460 (601h30m47.087576495s) [noise=true , address=145.79.227.5:4222 , piece_id=LA4M42KLBEDPYALFRQHFTE2C6VQ5R6LW6QBZARO6U2ZDRAJPLUEA, seg_pos={0 0}, upload=527201]: Uploading piece...
Dec 30 16:59:47.217694005 (601h30m47.08877704s) [noise=true , address=45.91.116.11:4118 , piece_id=2BJOAC3WIGLSXPVXGDN4ORGVENIRZ6ZNFI4GT2SROQPETNQNXHQQ, seg_pos={0 0}, upload=527201]: Uploading piece...
Dec 30 16:59:47.218989001 (601h30m47.090072036s) [noise=true , address=145.79.227.42:4123 , piece_id=NQBH6QQPUNNDA3CCSAOITPVQ3SAO77E42W2FDBPF4GRTYD6VSWFQ, seg_pos={0 0}, upload=527201]: Uploading piece...
Dec 30 16:59:47.220326387 (601h30m47.091409432s) [noise=true , address=145.79.227.31:4797 , piece_id=M4MB7O3EA3NDFGKYBKZOPLM7JDU4IXP6HG4XHL72O4GJKSQFTKAQ, seg_pos={0 0}, upload=527201]: Uploading piece...
Dec 30 16:59:47.221645088 (601h30m47.092728123s) [noise=true , address=145.79.227.6:4259 , piece_id=RM4WPRLPBQJRA2B2AR6CDZXO7QBFASUXGCP2NDOZI6LWM4HTHB2A, seg_pos={0 0}, upload=527201]: Uploading piece...
Dec 30 16:59:47.222831628 (601h30m47.093914663s) [noise=true , address=77.90.31.19:4841 , piece_id=5CVPDBWZZMTF4RU7YLDDI4HJ57H7RMGILPX7PIMFPTNNC4MFKQEQ, seg_pos={0 0}, upload=527201]: Uploading piece...
Dec 30 16:59:48.903207088 (601h30m48.774290123s) [upload=527201]: Next returned nil segment
Dec 30 16:59:48.903238007 (601h30m48.774321042s) [upload=527201]: Waiting for error group managing segments...
Dec 30 16:59:49.187976684 (601h30m49.059059729s) [noise=true , address=145.79.227.45:4230 , piece_id=4IXXDMMMJ5Q3KUKH26EQWS5DXHNIFC3PI33QFPYJB6677KNTT5OQ, seg_pos={0 0}, upload=527201]: Done uploading piece. err: <nil>
Dec 30 16:59:49.198069342 (601h30m49.069152377s) [noise=true , address=145.79.227.50:4373 , piece_id=MQF3YUF7U5BK5ZJGSHRCDHWNB4ZMIBWYLQA2XNQHLMUR63L2BBQA, seg_pos={0 0}, upload=527201]: Done uploading piece. err: <nil>
Dec 30 16:59:49.202160432 (601h30m49.073243467s) [noise=true , address=145.79.227.48:4302 , piece_id=DGZIHINKA2M3W7L4MVS7ITGI6W2UBSOMZGKZTXXXMD2HVY3HLFOA, seg_pos={0 0}, upload=527201]: Done uploading piece. err: <nil>
Dec 30 16:59:49.213483222 (601h30m49.084566257s) [noise=true , address=145.79.227.42:4123 , piece_id=NQBH6QQPUNNDA3CCSAOITPVQ3SAO77E42W2FDBPF4GRTYD6VSWFQ, seg_pos={0 0}, upload=527201]: Done uploading piece. err: <nil>
Dec 30 16:59:49.229077405 (601h30m49.10016045s) [noise=true , address=145.79.227.31:4797 , piece_id=M4MB7O3EA3NDFGKYBKZOPLM7JDU4IXP6HG4XHL72O4GJKSQFTKAQ, seg_pos={0 0}, upload=527201]: Done uploading piece. err: <nil>
Dec 30 16:59:49.232909141 (601h30m49.103992176s) [noise=true , address=145.79.227.47:4267 , piece_id=UMVLUUZQXRFNQL7UJLNSVWDXXBIMQNQYEM2EDBW4RYL7XWVNZMVA, seg_pos={0 0}, upload=527201]: Done uploading piece. err: <nil>
Dec 30 16:59:49.233336004 (601h30m49.104419039s) [noise=true , address=145.79.227.18:4007 , piece_id=IXTWS7425QTXR65NJANPFYJOKLP4KCNTSY6JPQWXV2STG7VLNPTQ, seg_pos={0 0}, upload=527201]: Done uploading piece. err: <nil>
Dec 30 16:59:49.291279996 (601h30m49.162363041s) [noise=true , address=145.79.227.5:4222 , piece_id=LA4M42KLBEDPYALFRQHFTE2C6VQ5R6LW6QBZARO6U2ZDRAJPLUEA, seg_pos={0 0}, upload=527201]: Done uploading piece. err: <nil>
Dec 30 16:59:49.301745013 (601h30m49.172828048s) [noise=true , address=145.79.227.6:4259 , piece_id=RM4WPRLPBQJRA2B2AR6CDZXO7QBFASUXGCP2NDOZI6LWM4HTHB2A, seg_pos={0 0}, upload=527201]: Done uploading piece. err: <nil>
Dec 30 16:59:49.373505102 (601h30m49.244588137s) [noise=true , address=145.79.227.23:4545 , piece_id=ZUTQSU7R3CIGTTOEK7YETJZDFCBLZUUCMAUUELTYQOHRGMUR5U7Q, seg_pos={0 0}, upload=527201]: Done uploading piece. err: <nil>
Dec 30 16:59:49.537360624 (601h30m49.408443659s) [noise=true , address=45.91.116.8:4046 , piece_id=WKQBEPCIFJKI6D733IDPXL3UM6NGOKEVS6XM5FIWSJCPZKNQZ62A, seg_pos={0 0}, upload=527201]: Done uploading piece. err: <nil>
Dec 30 16:59:49.634981796 (601h30m49.506064831s) [noise=true , address=45.91.116.6:4976 , piece_id=X6C732MSJMQNWENN6RLVD62DHRDN3PJALVUH37L5NGBVWVDZVYUQ, seg_pos={0 0}, upload=527201]: Done uploading piece. err: <nil>
Dec 30 16:59:49.791539009 (601h30m49.662622044s) [noise=true , address=77.90.31.19:4841 , piece_id=5CVPDBWZZMTF4RU7YLDDI4HJ57H7RMGILPX7PIMFPTNNC4MFKQEQ, seg_pos={0 0}, upload=527201]: Done uploading piece. err: <nil>
Dec 30 16:59:49.791566211 (601h30m49.662649257s) [seg_pos={0 0}, upload=527201]: Segment reached optimal threshold of 13 pieces.
Dec 30 16:59:49.791768326 (601h30m49.662851371s) [noise=true , address=145.79.227.40:4049 , piece_id=UIBS7DURG2M5ARTDGHIOB66GL7GJFH2VDNFHTOHKFCQVTVDTHDVQ, seg_pos={0 0}, upload=527201]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF
Dec 30 16:59:49.791776422 (601h30m49.662859457s) [noise=true , address=145.79.227.40:4049 , piece_id=UIBS7DURG2M5ARTDGHIOB66GL7GJFH2VDNFHTOHKFCQVTVDTHDVQ, seg_pos={0 0}, upload=527201]: Long tail cancellation due to optimal threshold
Dec 30 16:59:49.791794476 (601h30m49.662877511s) [noise=true , address=45.91.116.11:4118 , piece_id=2BJOAC3WIGLSXPVXGDN4ORGVENIRZ6ZNFI4GT2SROQPETNQNXHQQ, seg_pos={0 0}, upload=527201]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p): protocol: expected piece hash; context canceled; EOF
Dec 30 16:59:49.791802100 (601h30m49.662885135s) [noise=true , address=45.91.116.11:4118 , piece_id=2BJOAC3WIGLSXPVXGDN4ORGVENIRZ6ZNFI4GT2SROQPETNQNXHQQ, seg_pos={0 0}, upload=527201]: Long tail cancellation due to optimal threshold
Dec 30 16:59:49.792001189 (601h30m49.663084224s) [noise=true , address=77.90.31.18:4805 , piece_id=SKRCNADR3PEBW6IZS4UK42PORHUNLSWF3YCRNT7YT5QVDAYATWEQ, seg_pos={0 0}, upload=527201]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: write tcp 188.166.225.170:53050->77.90.31.18:4805: use of closed network connection; write tcp 188.166.225.170:53050->77.90.31.18:4805: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53050->77.90.31.18:4805: use of closed network connection
Dec 30 16:59:49.792007641 (601h30m49.663090686s) [noise=true , address=77.90.31.18:4805 , piece_id=SKRCNADR3PEBW6IZS4UK42PORHUNLSWF3YCRNT7YT5QVDAYATWEQ, seg_pos={0 0}, upload=527201]: Long tail cancellation due to optimal threshold
Dec 30 16:59:49.792022360 (601h30m49.663105395s) [seg_pos={0 0}, upload=527201]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 16:59:49.792089899 (601h30m49.663172934s) [upload=527201]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 16:59:49.792096441 (601h30m49.663179476s) [upload=527201]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 16:59:50.076398858 (601h30m49.947481913s) [upload=527201]: Done uploading segments...
Dec 30 16:59:50.076445606 (601h30m49.947528651s) [upload=527201]: Upload finished. err: <nil>
Duration: 6.057605915s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [..................X.............X.......X...................] duration:
28.106188827s trace:
https://jaeger.dev.storj.io/trace/3f4379d3637c8e4b
FAIL (2025-12-30 17:00:35 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1WBuBdGxdPXi6VsNf4FJpn3LpWoQztM7CV6YjAW1mUDVjAUjn2, Piece ID: MS4HNXUQIITLBMEHZDPJMEH7FSLMBBGKJSIP5BY2BDUF3IE2NKIQ) context deadline exceeded; piecestore: (Node ID: 1soiTQsiMNe5BkNHwwAGykmtP9LtLxncqE7Tv728ak8pjFeshg, Piece ID: BHVQNSBU3LP35Z5WQYTRPJZNNWTDZKWYK2CMRLHSOGBJ4IXGZ5SA) context deadline exceeded; piecestore: (Node ID: 1548pzNGffaMonic7dMK5VtZhLHtae3iffd9wjfjyTCp4XJ7Lo, Piece ID: MVRLWV5URMPV57R52J426SXY5ZWQGHOTL42GO2GPEZ4LA7B2JGCQ) context deadline exceeded
======LOGS======
5m56.099625936s) [seg_pos={0 0}, upload=526987]: Begin upload segment...
Dec 30 16:44:56.228605921 (601h15m56.099688967s) [seg_pos={0 0}, upload=526987]: Done begin upload segment.
Dec 30 16:44:56.228736641 (601h15m56.099819686s) [noise=true , address=145.79.227.34:4923 , piece_id=OB3KQEHYXE4BL73EDDQDN2LSVYJYYQQFUSFJ655BFJISLBGOH4XQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.228753061 (601h15m56.099836117s) [noise=true , address=64.49.13.26:4947 , piece_id=BHVQNSBU3LP35Z5WQYTRPJZNNWTDZKWYK2CMRLHSOGBJ4IXGZ5SA, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.229701116 (601h15m56.100784151s) [noise=true , address=45.91.116.8:4071 , piece_id=3PUQCDJAXSQGEM47BTD2B5HWWNK3PUS6AEVQ63BEC2Q2CKCB4PXQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.230399837 (601h15m56.101482882s) [noise=true , address=77.90.31.21:4899 , piece_id=DCYPDKTC4S5LG3XEBQWZT3HGFJ63T3UC3PFUDLF7OOJHNMYW4KRQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.230834926 (601h15m56.101917971s) [noise=true , address=77.90.31.19:4853 , piece_id=MS4HNXUQIITLBMEHZDPJMEH7FSLMBBGKJSIP5BY2BDUF3IE2NKIQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.232511007 (601h15m56.103594112s) [noise=true , address=145.79.227.8:4344 , piece_id=6L5DD7CQJ5JZ7IPFX2IGHNEP7M6BZU2DA5Z5GEOKGPAMVZUXQHOQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.233047428 (601h15m56.104130473s) [noise=true , address=145.79.227.31:4808 , piece_id=MW4UFSJ4XFLXK7ZLWZFWTP2N4QQUS24WP3O44DJWT7AK2WCWAT3Q, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.233557500 (601h15m56.104640545s) [noise=true , address=145.79.227.33:4903 , piece_id=THSFTSO2Y4QVWIGNUOU4AKFGTSQWQ6FLNIIDISIVNRF7XZZWWPCQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.234065937 (601h15m56.105148983s) [noise=true , address=64.49.13.137:4493 , piece_id=QZSPIHMAGKSHSAJYJ6BR26EMTCWFAEPOPC4O65YN6SL6222H4HWQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.234513860 (601h15m56.105596896s) [noise=true , address=64.49.13.27:4965 , piece_id=AQTJD7BXKPMIAGEOGJCY5Y64IVU2SHCZEFJL2NLQAHVX5HR33N7Q, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.236125519 (601h15m56.107208554s) [noise=true , address=77.90.31.11:4567 , piece_id=NMCHY46TZ6W6EPVLVVCGQXHFERHBLSFB3J55AD32KQSKOVXABO7A, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.237179756 (601h15m56.108262791s) [noise=true , address=145.79.227.40:4064 , piece_id=SEYSDBZHX6SHMFZBGY3XDIYEOTWFCNGG2KNNTCIXYQRZP7FZOVJQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.238439144 (601h15m56.109522189s) [noise=true , address=145.79.227.30:4790 , piece_id=5VRI4FYK7J54TCOY7DX6TCQ3634N3TATGDZUZDWRBBLMXCGRVR3Q, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.239316585 (601h15m56.11039962s) [noise=true , address=145.79.227.44:4212 , piece_id=ZV3XMTVGJR2D5X4EEGGDTGGGXSQI4QNCJZCI3GPR6RTD7JEQWLGA, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.242138468 (601h15m56.113221503s) [noise=true , address=45.91.116.11:4131 , piece_id=YAFIDLBM762GWTNKDW62FMNLE4YHCNYW2WU3R4AMO5NELMVOHHHQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.243965547 (601h15m56.115048582s) [noise=true , address=77.90.31.18:4829 , piece_id=MVRLWV5URMPV57R52J426SXY5ZWQGHOTL42GO2GPEZ4LA7B2JGCQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.250805150 (601h15m56.121888185s) [noise=true , address=5.231.73.4:4033 , piece_id=CCH6AH47YBQPGPF4MPZU2NPT7HPF4D3KT3ZVDITRYF7UW5JUKWDQ, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:56.254973396 (601h15m56.126056431s) [noise=true , address=145.79.227.49:4341 , piece_id=6GKQQIBOECLDEEDZNO5AYKX7XMXZYUA6WNGKD3FQSF2Q4O6A4MQA, seg_pos={0 0}, upload=526987]: Uploading piece...
Dec 30 16:44:57.691193696 (601h15m57.562276731s) [upload=526987]: Next returned nil segment
Dec 30 16:44:57.691206460 (601h15m57.562289495s) [upload=526987]: Waiting for error group managing segments...
Dec 30 16:44:58.085111563 (601h15m57.956194598s) [noise=true , address=64.49.13.137:4493 , piece_id=QZSPIHMAGKSHSAJYJ6BR26EMTCWFAEPOPC4O65YN6SL6222H4HWQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.086614434 (601h15m57.957697479s) [noise=true , address=64.49.13.27:4965 , piece_id=AQTJD7BXKPMIAGEOGJCY5Y64IVU2SHCZEFJL2NLQAHVX5HR33N7Q, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.119421281 (601h15m57.990504316s) [noise=true , address=145.79.227.8:4344 , piece_id=6L5DD7CQJ5JZ7IPFX2IGHNEP7M6BZU2DA5Z5GEOKGPAMVZUXQHOQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.125317667 (601h15m57.996400702s) [noise=true , address=145.79.227.31:4808 , piece_id=MW4UFSJ4XFLXK7ZLWZFWTP2N4QQUS24WP3O44DJWT7AK2WCWAT3Q, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.262371457 (601h15m58.133454502s) [noise=true , address=64.49.13.26:4947 , piece_id=BHVQNSBU3LP35Z5WQYTRPJZNNWTDZKWYK2CMRLHSOGBJ4IXGZ5SA, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.284980216 (601h15m58.156063251s) [noise=true , address=77.90.31.19:4853 , piece_id=MS4HNXUQIITLBMEHZDPJMEH7FSLMBBGKJSIP5BY2BDUF3IE2NKIQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.387549123 (601h15m58.258632158s) [noise=true , address=77.90.31.18:4829 , piece_id=MVRLWV5URMPV57R52J426SXY5ZWQGHOTL42GO2GPEZ4LA7B2JGCQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.394280961 (601h15m58.265364006s) [noise=true , address=5.231.73.4:4033 , piece_id=CCH6AH47YBQPGPF4MPZU2NPT7HPF4D3KT3ZVDITRYF7UW5JUKWDQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.510328564 (601h15m58.381411609s) [noise=true , address=45.91.116.8:4071 , piece_id=3PUQCDJAXSQGEM47BTD2B5HWWNK3PUS6AEVQ63BEC2Q2CKCB4PXQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.545484694 (601h15m58.416567739s) [noise=true , address=45.91.116.11:4131 , piece_id=YAFIDLBM762GWTNKDW62FMNLE4YHCNYW2WU3R4AMO5NELMVOHHHQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.710409562 (601h15m58.581492607s) [noise=true , address=77.90.31.21:4899 , piece_id=DCYPDKTC4S5LG3XEBQWZT3HGFJ63T3UC3PFUDLF7OOJHNMYW4KRQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.981893937 (601h15m58.852976982s) [noise=true , address=145.79.227.44:4212 , piece_id=ZV3XMTVGJR2D5X4EEGGDTGGGXSQI4QNCJZCI3GPR6RTD7JEQWLGA, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:58.995663946 (601h15m58.866746991s) [noise=true , address=145.79.227.49:4341 , piece_id=6GKQQIBOECLDEEDZNO5AYKX7XMXZYUA6WNGKD3FQSF2Q4O6A4MQA, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:59.009469091 (601h15m58.880552136s) [noise=true , address=145.79.227.40:4064 , piece_id=SEYSDBZHX6SHMFZBGY3XDIYEOTWFCNGG2KNNTCIXYQRZP7FZOVJQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:59.009489950 (601h15m58.880572986s) [seg_pos={0 0}, upload=526987]: Segment reached optimal threshold of 14 pieces.
Dec 30 16:44:59.009662329 (601h15m58.880745374s) [noise=true , address=145.79.227.34:4923 , piece_id=OB3KQEHYXE4BL73EDDQDN2LSVYJYYQQFUSFJ655BFJISLBGOH4XQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VXAxRGiJQor24W7DYSeUQUu1QF7EcX6BQYgfUheACRs2quFFu): protocol: expected piece hash; context canceled; EOF
Dec 30 16:44:59.009668390 (601h15m58.880751425s) [noise=true , address=145.79.227.34:4923 , piece_id=OB3KQEHYXE4BL73EDDQDN2LSVYJYYQQFUSFJ655BFJISLBGOH4XQ, seg_pos={0 0}, upload=526987]: Long tail cancellation due to optimal threshold
Dec 30 16:44:59.009857671 (601h15m58.880940706s) [noise=true , address=145.79.227.33:4903 , piece_id=THSFTSO2Y4QVWIGNUOU4AKFGTSQWQ6FLNIIDISIVNRF7XZZWWPCQ, seg_pos={0 0}, upload=526987]: Done uploading piece. err: <nil>
Dec 30 16:44:59.009921232 (601h15m58.881004267s) [noise=true , address=145.79.227.30:4790 , piece_id=5VRI4FYK7J54TCOY7DX6TCQ3634N3TATGDZUZDWRBBLMXCGRVR3Q, seg_pos={0 0}, upload=526987]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12kuSMof8EbvsaMCdVxKWjgTT5TfLJnTAeHUf8DKNC3UGQn5q7T): protocol: expected piece hash; context canceled; EOF
Dec 30 16:44:59.009927314 (601h15m58.881010359s) [noise=true , address=145.79.227.30:4790 , piece_id=5VRI4FYK7J54TCOY7DX6TCQ3634N3TATGDZUZDWRBBLMXCGRVR3Q, seg_pos={0 0}, upload=526987]: Long tail cancellation due to optimal threshold
Dec 30 16:44:59.010092589 (601h15m58.881175624s) [noise=true , address=77.90.31.11:4567 , piece_id=NMCHY46TZ6W6EPVLVVCGQXHFERHBLSFB3J55AD32KQSKOVXABO7A, seg_pos={0 0}, upload=526987]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12HJVmEuxjrwWCcdwxWYFC5fTULds1kiRH82DiRWipWhRT1Qwdk): protocol: expected piece hash; context canceled; EOF
Dec 30 16:44:59.010101946 (601h15m58.881184981s) [noise=true , address=77.90.31.11:4567 , piece_id=NMCHY46TZ6W6EPVLVVCGQXHFERHBLSFB3J55AD32KQSKOVXABO7A, seg_pos={0 0}, upload=526987]: Long tail cancellation due to optimal threshold
Dec 30 16:44:59.010125410 (601h15m58.881208445s) [seg_pos={0 0}, upload=526987]: Done waiting for segment. successful: 15 optimal: 14 errs: <nil>
Dec 30 16:44:59.010204091 (601h15m58.881287136s) [upload=526987]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 16:44:59.010211274 (601h15m58.881294339s) [upload=526987]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 16:44:59.309660956 (601h15m59.180744001s) [upload=526987]: Done uploading segments...
Dec 30 16:44:59.309679662 (601h15m59.180762697s) [upload=526987]: Upload finished. err: <nil>
Duration: 28.106209297s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
20 out of 60 is failed (threshold=6) [X...XXX.X.X.X...XXX..X....XXX..X...XX......X.XX.............] duration:
3.629790111s trace:
https://jaeger.dev.storj.io/trace/2400bb34adb4564c
FAIL (2025-12-30 17:00:26 UTC)
orbiter warning: context deadline exceeded
Duration: 3.629817371s
core
uplink
us-select-1
us1-placement-20
|