Orbiter History

Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 356874 reports (Page 222 of 7138)

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 42 out of 60 is failed (threshold=6) [X.XX.X.X.X.XXXXXX.XXXX...XX.X.XXXXXXXXX.XXXXXXX...XXXXX.XX..] duration: 3.571440548s trace: https://jaeger.dev.storj.io/trace/3532402db8c825 FAIL (2025-12-30 15:59:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.57156205s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 11.864370442s trace: https://jaeger.dev.storj.io/trace/547648a2cfce50e7 FAIL (2025-12-30 15:58:52 UTC) orbiter warning: context deadline exceeded Duration: 11.864393126s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X............................X..XX..X..................X...] duration: 35.096817257s trace: https://jaeger.dev.storj.io/trace/2ead4e08bdf51a6a FAIL (2025-12-30 15:58:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 43:4696 , piece_id=3WM5R3KUHXN4WVDOKBAOFX7RAI6ZUTV7UKHUOYAKHUYPM4M24N2A, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.239224572 (600h25m38.110307617s) [noise=true , address=45.91.116.12:4241 , piece_id=FYZGK7TFHSQ5BDLEMJIOGXWVWSZ45NRVSYQ4DT6DHBANOCBOZZWA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.239811830 (600h25m38.110894865s) [noise=true , address=145.79.227.43:4189 , piece_id=DYHLPLLBNG6B3FXERO33ZDDSHFUHQIIYGQZJFE2UMJXMLUBYC3KQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.241164706 (600h25m38.112247741s) [noise=true , address=77.90.31.12:4604 , piece_id=OKY6LAK735LFQ4MDC5K6LJPFQITQSGNOZUGNU6US25RELWLK33BQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.244508833 (600h25m38.115591878s) [noise=true , address=145.79.227.49:4350 , piece_id=MA66IA2WTN55SFQDQ4LF7H5LNVUK3IMIHHRUH4TXWHWZPTQDBDCA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.244943361 (600h25m38.116026396s) [noise=true , address=77.90.31.17:4800 , piece_id=K3HNTJHOSIIUBSKISKWG5YRGLKNKHDOWHITXEDDLKPATPIC4IKTQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.250127493 (600h25m38.121210528s) [noise=true , address=145.79.227.35:4516 , piece_id=RDI4UFYB6YC22A5QOGDTLRGCG63TSV6XJFKGLKGSJBC5CH3KCQMQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.250776008 (600h25m38.121859053s) [noise=true , address=145.79.227.39:4033 , piece_id=BLTCPLYP5MIQO5Y7WWPT5QIDV6QM3YWAFMLOEWFOVWFLZOO67QSA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.253191467 (600h25m38.124274512s) [noise=true , address=145.79.227.44:4196 , piece_id=ZUDLORSKFXXIGHZ3YNIE3T7JF7OG55NZKPTSFOUVHAISGF7V7J5Q, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.254554973 (600h25m38.125638018s) [noise=true , address=45.91.116.5:4314 , piece_id=XWWPTNVPYYZ6VL26MACQEJTB2FYN6523YJGZK4I5B4OFRJNUMY3A, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.255815333 (600h25m38.126898378s) [noise=true , address=145.79.227.40:4077 , piece_id=AWXM3BGQVCAARS5F5UISWWIFAVWLWV2KML4FAP6BQ6QNXJTG77NA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.259122731 (600h25m38.130205776s) [noise=true , address=64.49.13.169:4713 , piece_id=KI4XABHC36ADSOANW3HVM3TCWZ4ZXXXNP7MWJVESDKD5WO26DJHQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.265517899 (600h25m38.136600944s) [noise=true , address=77.90.31.19:4854 , piece_id=4W27FHSBO3L2TSAZL2JFCTUPHYV7RI5XDQ4SJWOCRCD5V4K4IK2Q, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.266738002 (600h25m38.137821047s) [noise=true , address=145.79.227.33:4877 , piece_id=GPUOFM2E3KZGDCEZQYPKADLJFDML5DS7OTWFW7AFB5DYAHBVBQ6Q, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.267328907 (600h25m38.138411952s) [noise=true , address=77.90.31.21:4911 , piece_id=B25VZGBKAGIB4ZN6OICFQFZTRQWZIIYDBFUWVNAW4QW26AVYSRPA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:39.692439689 (600h25m39.563522734s) [upload=526249]: Next returned nil segment Dec 30 15:54:39.692451583 (600h25m39.563534618s) [upload=526249]: Waiting for error group managing segments... Dec 30 15:54:40.135912060 (600h25m40.006995095s) [noise=true , address=145.79.227.39:4033 , piece_id=BLTCPLYP5MIQO5Y7WWPT5QIDV6QM3YWAFMLOEWFOVWFLZOO67QSA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.285764442 (600h25m40.156847487s) [noise=true , address=5.231.73.3:4032 , piece_id=7MBVEHMVMT6XTZJZHHWA65RMFNIB7EH4HU63WTLBPPB6CILGMYAA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.400071651 (600h25m40.271154696s) [noise=true , address=77.90.31.12:4604 , piece_id=OKY6LAK735LFQ4MDC5K6LJPFQITQSGNOZUGNU6US25RELWLK33BQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.423162570 (600h25m40.294245605s) [noise=true , address=45.91.116.12:4241 , piece_id=FYZGK7TFHSQ5BDLEMJIOGXWVWSZ45NRVSYQ4DT6DHBANOCBOZZWA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.428962032 (600h25m40.300045067s) [noise=true , address=77.90.31.17:4800 , piece_id=K3HNTJHOSIIUBSKISKWG5YRGLKNKHDOWHITXEDDLKPATPIC4IKTQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.543712124 (600h25m40.414795169s) [noise=true , address=45.91.116.5:4314 , piece_id=XWWPTNVPYYZ6VL26MACQEJTB2FYN6523YJGZK4I5B4OFRJNUMY3A, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.993222051 (600h25m40.864305086s) [noise=true , address=145.79.227.44:4196 , piece_id=ZUDLORSKFXXIGHZ3YNIE3T7JF7OG55NZKPTSFOUVHAISGF7V7J5Q, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.006811979 (600h25m40.877895024s) [noise=true , address=193.149.164.143:4696 , piece_id=3WM5R3KUHXN4WVDOKBAOFX7RAI6ZUTV7UKHUOYAKHUYPM4M24N2A, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.024553631 (600h25m40.895636666s) [noise=true , address=145.79.227.40:4077 , piece_id=AWXM3BGQVCAARS5F5UISWWIFAVWLWV2KML4FAP6BQ6QNXJTG77NA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.054002647 (600h25m40.925085692s) [noise=true , address=145.79.227.49:4350 , piece_id=MA66IA2WTN55SFQDQ4LF7H5LNVUK3IMIHHRUH4TXWHWZPTQDBDCA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.074754852 (600h25m40.945837897s) [noise=true , address=77.90.31.21:4911 , piece_id=B25VZGBKAGIB4ZN6OICFQFZTRQWZIIYDBFUWVNAW4QW26AVYSRPA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.076478905 (600h25m40.94756195s) [noise=true , address=145.79.227.35:4516 , piece_id=RDI4UFYB6YC22A5QOGDTLRGCG63TSV6XJFKGLKGSJBC5CH3KCQMQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.091294145 (600h25m40.96237718s) [noise=true , address=145.79.227.30:4789 , piece_id=5JZFEA3P6EICC2MGUYOQWY4OWOVHXWZLBJBBSVUAMPLV2EHZOZLA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.138722376 (600h25m41.009805421s) [noise=true , address=145.79.227.49:4359 , piece_id=MCTYZULSW2Z2A6HXPREG3GOEYF5QLBPKEB7DDGT32FNTWHC646CQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.138745481 (600h25m41.009828516s) [seg_pos={0 0}, upload=526249]: Segment reached optimal threshold of 14 pieces. Dec 30 15:54:41.138965780 (600h25m41.010048815s) [noise=true , address=145.79.227.33:4877 , piece_id=GPUOFM2E3KZGDCEZQYPKADLJFDML5DS7OTWFW7AFB5DYAHBVBQ6Q, seg_pos={0 0}, upload=526249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17TuauVrjKmehRpXqmK7bemU9V33GDZu9RGBs6fanXvJkV1gdH): protocol: expected piece hash; context canceled; EOF Dec 30 15:54:41.138974496 (600h25m41.010057531s) [noise=true , address=145.79.227.33:4877 , piece_id=GPUOFM2E3KZGDCEZQYPKADLJFDML5DS7OTWFW7AFB5DYAHBVBQ6Q, seg_pos={0 0}, upload=526249]: Long tail cancellation due to optimal threshold Dec 30 15:54:41.139084886 (600h25m41.010167921s) [noise=true , address=77.90.31.19:4854 , piece_id=4W27FHSBO3L2TSAZL2JFCTUPHYV7RI5XDQ4SJWOCRCD5V4K4IK2Q, seg_pos={0 0}, upload=526249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16HwqYhJnhseWEhzzv4QreZ953rB5cKxqFX8swEqaqiRXCRsAE): protocol: write tcp 188.166.225.170:48144->77.90.31.19:4854: use of closed network connection; write tcp 188.166.225.170:48144->77.90.31.19:4854: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48144->77.90.31.19:4854: use of closed network connection Dec 30 15:54:41.139093533 (600h25m41.010176568s) [noise=true , address=77.90.31.19:4854 , piece_id=4W27FHSBO3L2TSAZL2JFCTUPHYV7RI5XDQ4SJWOCRCD5V4K4IK2Q, seg_pos={0 0}, upload=526249]: Long tail cancellation due to optimal threshold Dec 30 15:54:41.139129352 (600h25m41.010212397s) [noise=true , address=145.79.227.43:4189 , piece_id=DYHLPLLBNG6B3FXERO33ZDDSHFUHQIIYGQZJFE2UMJXMLUBYC3KQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7): protocol: expected piece hash; context canceled; EOF Dec 30 15:54:41.139138429 (600h25m41.010221484s) [noise=true , address=145.79.227.43:4189 , piece_id=DYHLPLLBNG6B3FXERO33ZDDSHFUHQIIYGQZJFE2UMJXMLUBYC3KQ, seg_pos={0 0}, upload=526249]: Long tail cancellation due to optimal threshold Dec 30 15:54:41.139238890 (600h25m41.010321925s) [noise=true , address=64.49.13.169:4713 , piece_id=KI4XABHC36ADSOANW3HVM3TCWZ4ZXXXNP7MWJVESDKD5WO26DJHQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:35558->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:35558->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35558->64.49.13.169:4713: use of closed network connection Dec 30 15:54:41.139244420 (600h25m41.010327455s) [noise=true , address=64.49.13.169:4713 , piece_id=KI4XABHC36ADSOANW3HVM3TCWZ4ZXXXNP7MWJVESDKD5WO26DJHQ, seg_pos={0 0}, upload=526249]: Long tail cancellation due to optimal threshold Dec 30 15:54:41.139323681 (600h25m41.010406716s) [seg_pos={0 0}, upload=526249]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 15:54:41.139392121 (600h25m41.010475166s) [upload=526249]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 15:54:41.139396971 (600h25m41.010480006s) [upload=526249]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 15:54:41.438011559 (600h25m41.309094604s) [upload=526249]: Done uploading segments... Dec 30 15:54:41.438036206 (600h25m41.309119251s) [upload=526249]: Upload finished. err: <nil> Duration: 35.096843327s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 43 out of 60 is failed (threshold=6) [XX.XX.X.X.X.XXXXXX.XXXX...XX.X.XXXXXXXXX.XXXXXXX...XXXXX.XX.] duration: 3.621621356s trace: https://jaeger.dev.storj.io/trace/3532402db8c825 FAIL (2025-12-30 15:58:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.621637648s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 11.242636727s trace: https://jaeger.dev.storj.io/trace/547648a2cfce50e7 FAIL (2025-12-30 15:58:28 UTC) orbiter warning: context deadline exceeded Duration: 11.242664179s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..X.X..........XX.......XX.XX..X....X.......X......X.X......] duration: 6.069129515s trace: https://jaeger.dev.storj.io/trace/54fff38414d25b0f FAIL (2025-12-30 15:58:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: AQGHSCVDMOODBX225O5SSOLNLJQYPAY2X7EJ7G5W6RP42ZJ4DGCQ) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: ZTXP5ESUQMWFEWTQUX2SMER7W2NJG5IISRWFAM3IKAKQFKJOGWSQ) context deadline exceeded ======LOGS====== Dec 30 15:53:33.440030169 (600h24m33.311113204s) [upload=526231]: Starting upload Dec 30 15:53:33.440113437 (600h24m33.311196492s) [upload=526231]: Done starting upload Dec 30 15:53:33.440168081 (600h24m33.311251116s) [upload=526231]: Uploading segments... Dec 30 15:53:33.440357231 (600h24m33.311440276s) [upload=526231]: Got next segment. Inline: false Dec 30 15:53:33.440411114 (600h24m33.311494159s) [upload=526231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 15:53:33.440416885 (600h24m33.31149992s) [upload=526231]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 15:53:33.684984430 (600h24m33.556067505s) [seg_pos={0 0}, upload=526231]: Begin upload segment... Dec 30 15:53:33.685033994 (600h24m33.556117029s) [seg_pos={0 0}, upload=526231]: Done begin upload segment. Dec 30 15:53:33.685081235 (600h24m33.55616427s) [noise=true , address=145.79.227.36:4519 , piece_id=5JAZZV45ZTNDSTJZIX2EI7RK7MCBXBQUV2PLSF2HBCRFCIJEA6OQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.685200161 (600h24m33.556283196s) [noise=true , address=145.79.227.23:4545 , piece_id=NHTKPJFEFGAJOEXUNJJ7TU6N27JJP3O5TFSMWF5I2SMGEYXWEAKQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.685709841 (600h24m33.556792887s) [noise=true , address=77.90.31.14:4664 , piece_id=YLXNEMZBXUYTFQ4P6ZFHPA6BIUH4TZ2OBXEQAJWYSCUX333L777A, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686084574 (600h24m33.557167619s) [noise=true , address=45.91.116.12:4227 , piece_id=5GQIOQC4VDIF3Z6J4TUPZG3XUC3AACZ53CPYB4JIBQD7F5BK2IFA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686094794 (600h24m33.55717783s) [noise=true , address=77.90.31.10:4517 , piece_id=5Y32J2NPA664L4ZGRDDYYX66FPWND4DNZ5OZXAADJ3LOXK3IA5BA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686429431 (600h24m33.557512476s) [noise=true , address=45.91.116.13:4153 , piece_id=AQGHSCVDMOODBX225O5SSOLNLJQYPAY2X7EJ7G5W6RP42ZJ4DGCQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686489927 (600h24m33.557572962s) [noise=true , address=45.91.116.8:4047 , piece_id=TLKGZQ2VDRAGBZCULZ35ZR6GEJXQEE2HZ35Q52XBEO5C4FTRFICA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686910547 (600h24m33.557993582s) [noise=true , address=145.79.227.37:4567 , piece_id=WSHVGPU7L73OET2M6WT7VPHGVRQV6ZCJRPAW7AEXBUQDFWQBNAUA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686946536 (600h24m33.558029581s) [noise=true , address=145.79.227.14:4838 , piece_id=YB4VPIAQTBJ6G5XFXDVJ3OL6YFAWMEBHHS63SEDX5U4HYKXT3KJQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.687266755 (600h24m33.55834979s) [noise=true , address=145.79.227.6:4257 , piece_id=53MG7UDQR3MJYNMQXY3IOPPKOQEOI3KFUTGHQUEHOKCNG5NTAWXA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.687625419 (600h24m33.558708464s) [noise=true , address=45.91.116.11:4120 , piece_id=ZTXP5ESUQMWFEWTQUX2SMER7W2NJG5IISRWFAM3IKAKQFKJOGWSQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.688010913 (600h24m33.559093948s) [noise=true , address=145.79.227.19:4077 , piece_id=JEX52J3QTGUWMY3FO6BD5IJV5UPUUCKVV4MYZAOAU3BWMKFSN3KQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.688360248 (600h24m33.559443283s) [noise=true , address=145.79.227.16:4910 , piece_id=UHDFUXEQCMN73AUB7WEJBG5HY2G4E25MQGSBNK6AKHMQMDZCJYRA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.688673685 (600h24m33.55975672s) [noise=true , address=145.79.227.48:4303 , piece_id=IO7FUTELAP2L3VVZEQN6N6QG42WMAOS7H5JEPFAST5YEKMKHYLXA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.689057035 (600h24m33.56014007s) [noise=true , address=145.79.227.2:4114 , piece_id=3MJV2XXBY776ZUIO2KU4MEP37SP6P2NYFO255DCI6IFUZWUAGOWQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.690890025 (600h24m33.56197308s) [noise=true , address=145.79.227.3:4150 , piece_id=BUBUCPH3LTJO2CR3IVIXEOJOVS422KQT6L2Z22OBZP6DFPCK5KAA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:35.401137462 (600h24m35.272220507s) [upload=526231]: Next returned nil segment Dec 30 15:53:35.401154925 (600h24m35.27223797s) [upload=526231]: Waiting for error group managing segments... Dec 30 15:53:35.673442202 (600h24m35.544525237s) [noise=true , address=145.79.227.23:4545 , piece_id=NHTKPJFEFGAJOEXUNJJ7TU6N27JJP3O5TFSMWF5I2SMGEYXWEAKQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.674880822 (600h24m35.545963857s) [noise=true , address=145.79.227.2:4114 , piece_id=3MJV2XXBY776ZUIO2KU4MEP37SP6P2NYFO255DCI6IFUZWUAGOWQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.688801769 (600h24m35.559884814s) [noise=true , address=145.79.227.36:4519 , piece_id=5JAZZV45ZTNDSTJZIX2EI7RK7MCBXBQUV2PLSF2HBCRFCIJEA6OQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.689704678 (600h24m35.560787713s) [noise=true , address=145.79.227.6:4257 , piece_id=53MG7UDQR3MJYNMQXY3IOPPKOQEOI3KFUTGHQUEHOKCNG5NTAWXA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.690785346 (600h24m35.561868381s) [noise=true , address=145.79.227.14:4838 , piece_id=YB4VPIAQTBJ6G5XFXDVJ3OL6YFAWMEBHHS63SEDX5U4HYKXT3KJQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.692882529 (600h24m35.563965564s) [noise=true , address=145.79.227.37:4567 , piece_id=WSHVGPU7L73OET2M6WT7VPHGVRQV6ZCJRPAW7AEXBUQDFWQBNAUA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.696575461 (600h24m35.567658506s) [noise=true , address=145.79.227.16:4910 , piece_id=UHDFUXEQCMN73AUB7WEJBG5HY2G4E25MQGSBNK6AKHMQMDZCJYRA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.697864125 (600h24m35.56894716s) [noise=true , address=145.79.227.19:4077 , piece_id=JEX52J3QTGUWMY3FO6BD5IJV5UPUUCKVV4MYZAOAU3BWMKFSN3KQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.758530774 (600h24m35.629613819s) [noise=true , address=145.79.227.48:4303 , piece_id=IO7FUTELAP2L3VVZEQN6N6QG42WMAOS7H5JEPFAST5YEKMKHYLXA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.796785737 (600h24m35.667868772s) [noise=true , address=145.79.227.3:4150 , piece_id=BUBUCPH3LTJO2CR3IVIXEOJOVS422KQT6L2Z22OBZP6DFPCK5KAA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.029561278 (600h24m35.900644313s) [noise=true , address=77.90.31.10:4517 , piece_id=5Y32J2NPA664L4ZGRDDYYX66FPWND4DNZ5OZXAADJ3LOXK3IA5BA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.070210490 (600h24m35.941293536s) [noise=true , address=45.91.116.13:4153 , piece_id=AQGHSCVDMOODBX225O5SSOLNLJQYPAY2X7EJ7G5W6RP42ZJ4DGCQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.082591286 (600h24m35.953674331s) [noise=true , address=45.91.116.11:4120 , piece_id=ZTXP5ESUQMWFEWTQUX2SMER7W2NJG5IISRWFAM3IKAKQFKJOGWSQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.082610784 (600h24m35.953693819s) [seg_pos={0 0}, upload=526231]: Segment reached optimal threshold of 13 pieces. Dec 30 15:53:36.082858675 (600h24m35.953941711s) [noise=true , address=77.90.31.14:4664 , piece_id=YLXNEMZBXUYTFQ4P6ZFHPA6BIUH4TZ2OBXEQAJWYSCUX333L777A, seg_pos={0 0}, upload=526231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 30 15:53:36.082868053 (600h24m35.953951088s) [noise=true , address=77.90.31.14:4664 , piece_id=YLXNEMZBXUYTFQ4P6ZFHPA6BIUH4TZ2OBXEQAJWYSCUX333L777A, seg_pos={0 0}, upload=526231]: Long tail cancellation due to optimal threshold Dec 30 15:53:36.082953826 (600h24m35.954036871s) [noise=true , address=45.91.116.12:4227 , piece_id=5GQIOQC4VDIF3Z6J4TUPZG3XUC3AACZ53CPYB4JIBQD7F5BK2IFA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 30 15:53:36.082969977 (600h24m35.954053012s) [noise=true , address=45.91.116.12:4227 , piece_id=5GQIOQC4VDIF3Z6J4TUPZG3XUC3AACZ53CPYB4JIBQD7F5BK2IFA, seg_pos={0 0}, upload=526231]: Long tail cancellation due to optimal threshold Dec 30 15:53:36.083266191 (600h24m35.954349226s) [noise=true , address=45.91.116.8:4047 , piece_id=TLKGZQ2VDRAGBZCULZ35ZR6GEJXQEE2HZ35Q52XBEO5C4FTRFICA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: read tcp 188.166.225.170:42816->45.91.116.8:4047: use of closed network connection; context canceled Dec 30 15:53:36.083274006 (600h24m35.954357041s) [noise=true , address=45.91.116.8:4047 , piece_id=TLKGZQ2VDRAGBZCULZ35ZR6GEJXQEE2HZ35Q52XBEO5C4FTRFICA, seg_pos={0 0}, upload=526231]: Long tail cancellation due to optimal threshold Dec 30 15:53:36.083296048 (600h24m35.954379083s) [seg_pos={0 0}, upload=526231]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 15:53:36.083385679 (600h24m35.954468714s) [upload=526231]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 15:53:36.083392341 (600h24m35.954475376s) [upload=526231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 15:53:36.379743600 (600h24m36.250826635s) [upload=526231]: Done uploading segments... Dec 30 15:53:36.379758638 (600h24m36.250841673s) [upload=526231]: Upload finished. err: <nil> Duration: 6.069156667s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 44 out of 60 is failed (threshold=6) [XXX.XX.X.X.X.XXXXXX.XXXX...XX.X.XXXXXXXXX.XXXXXXX...XXXXX.XX] duration: 5.001920112s trace: https://jaeger.dev.storj.io/trace/3532402db8c825 FAIL (2025-12-30 15:58:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001943016s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...X.X..........XX.......XX.XX..X....X.......X......X.X.....] duration: 27.711652435s trace: https://jaeger.dev.storj.io/trace/54fff38414d25b0f FAIL (2025-12-30 15:57:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: AQGHSCVDMOODBX225O5SSOLNLJQYPAY2X7EJ7G5W6RP42ZJ4DGCQ) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: ZTXP5ESUQMWFEWTQUX2SMER7W2NJG5IISRWFAM3IKAKQFKJOGWSQ) context deadline exceeded ======LOGS====== Dec 30 15:53:33.440030169 (600h24m33.311113204s) [upload=526231]: Starting upload Dec 30 15:53:33.440113437 (600h24m33.311196492s) [upload=526231]: Done starting upload Dec 30 15:53:33.440168081 (600h24m33.311251116s) [upload=526231]: Uploading segments... Dec 30 15:53:33.440357231 (600h24m33.311440276s) [upload=526231]: Got next segment. Inline: false Dec 30 15:53:33.440411114 (600h24m33.311494159s) [upload=526231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 15:53:33.440416885 (600h24m33.31149992s) [upload=526231]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 15:53:33.684984430 (600h24m33.556067505s) [seg_pos={0 0}, upload=526231]: Begin upload segment... Dec 30 15:53:33.685033994 (600h24m33.556117029s) [seg_pos={0 0}, upload=526231]: Done begin upload segment. Dec 30 15:53:33.685081235 (600h24m33.55616427s) [noise=true , address=145.79.227.36:4519 , piece_id=5JAZZV45ZTNDSTJZIX2EI7RK7MCBXBQUV2PLSF2HBCRFCIJEA6OQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.685200161 (600h24m33.556283196s) [noise=true , address=145.79.227.23:4545 , piece_id=NHTKPJFEFGAJOEXUNJJ7TU6N27JJP3O5TFSMWF5I2SMGEYXWEAKQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.685709841 (600h24m33.556792887s) [noise=true , address=77.90.31.14:4664 , piece_id=YLXNEMZBXUYTFQ4P6ZFHPA6BIUH4TZ2OBXEQAJWYSCUX333L777A, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686084574 (600h24m33.557167619s) [noise=true , address=45.91.116.12:4227 , piece_id=5GQIOQC4VDIF3Z6J4TUPZG3XUC3AACZ53CPYB4JIBQD7F5BK2IFA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686094794 (600h24m33.55717783s) [noise=true , address=77.90.31.10:4517 , piece_id=5Y32J2NPA664L4ZGRDDYYX66FPWND4DNZ5OZXAADJ3LOXK3IA5BA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686429431 (600h24m33.557512476s) [noise=true , address=45.91.116.13:4153 , piece_id=AQGHSCVDMOODBX225O5SSOLNLJQYPAY2X7EJ7G5W6RP42ZJ4DGCQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686489927 (600h24m33.557572962s) [noise=true , address=45.91.116.8:4047 , piece_id=TLKGZQ2VDRAGBZCULZ35ZR6GEJXQEE2HZ35Q52XBEO5C4FTRFICA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686910547 (600h24m33.557993582s) [noise=true , address=145.79.227.37:4567 , piece_id=WSHVGPU7L73OET2M6WT7VPHGVRQV6ZCJRPAW7AEXBUQDFWQBNAUA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686946536 (600h24m33.558029581s) [noise=true , address=145.79.227.14:4838 , piece_id=YB4VPIAQTBJ6G5XFXDVJ3OL6YFAWMEBHHS63SEDX5U4HYKXT3KJQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.687266755 (600h24m33.55834979s) [noise=true , address=145.79.227.6:4257 , piece_id=53MG7UDQR3MJYNMQXY3IOPPKOQEOI3KFUTGHQUEHOKCNG5NTAWXA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.687625419 (600h24m33.558708464s) [noise=true , address=45.91.116.11:4120 , piece_id=ZTXP5ESUQMWFEWTQUX2SMER7W2NJG5IISRWFAM3IKAKQFKJOGWSQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.688010913 (600h24m33.559093948s) [noise=true , address=145.79.227.19:4077 , piece_id=JEX52J3QTGUWMY3FO6BD5IJV5UPUUCKVV4MYZAOAU3BWMKFSN3KQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.688360248 (600h24m33.559443283s) [noise=true , address=145.79.227.16:4910 , piece_id=UHDFUXEQCMN73AUB7WEJBG5HY2G4E25MQGSBNK6AKHMQMDZCJYRA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.688673685 (600h24m33.55975672s) [noise=true , address=145.79.227.48:4303 , piece_id=IO7FUTELAP2L3VVZEQN6N6QG42WMAOS7H5JEPFAST5YEKMKHYLXA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.689057035 (600h24m33.56014007s) [noise=true , address=145.79.227.2:4114 , piece_id=3MJV2XXBY776ZUIO2KU4MEP37SP6P2NYFO255DCI6IFUZWUAGOWQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.690890025 (600h24m33.56197308s) [noise=true , address=145.79.227.3:4150 , piece_id=BUBUCPH3LTJO2CR3IVIXEOJOVS422KQT6L2Z22OBZP6DFPCK5KAA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:35.401137462 (600h24m35.272220507s) [upload=526231]: Next returned nil segment Dec 30 15:53:35.401154925 (600h24m35.27223797s) [upload=526231]: Waiting for error group managing segments... Dec 30 15:53:35.673442202 (600h24m35.544525237s) [noise=true , address=145.79.227.23:4545 , piece_id=NHTKPJFEFGAJOEXUNJJ7TU6N27JJP3O5TFSMWF5I2SMGEYXWEAKQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.674880822 (600h24m35.545963857s) [noise=true , address=145.79.227.2:4114 , piece_id=3MJV2XXBY776ZUIO2KU4MEP37SP6P2NYFO255DCI6IFUZWUAGOWQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.688801769 (600h24m35.559884814s) [noise=true , address=145.79.227.36:4519 , piece_id=5JAZZV45ZTNDSTJZIX2EI7RK7MCBXBQUV2PLSF2HBCRFCIJEA6OQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.689704678 (600h24m35.560787713s) [noise=true , address=145.79.227.6:4257 , piece_id=53MG7UDQR3MJYNMQXY3IOPPKOQEOI3KFUTGHQUEHOKCNG5NTAWXA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.690785346 (600h24m35.561868381s) [noise=true , address=145.79.227.14:4838 , piece_id=YB4VPIAQTBJ6G5XFXDVJ3OL6YFAWMEBHHS63SEDX5U4HYKXT3KJQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.692882529 (600h24m35.563965564s) [noise=true , address=145.79.227.37:4567 , piece_id=WSHVGPU7L73OET2M6WT7VPHGVRQV6ZCJRPAW7AEXBUQDFWQBNAUA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.696575461 (600h24m35.567658506s) [noise=true , address=145.79.227.16:4910 , piece_id=UHDFUXEQCMN73AUB7WEJBG5HY2G4E25MQGSBNK6AKHMQMDZCJYRA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.697864125 (600h24m35.56894716s) [noise=true , address=145.79.227.19:4077 , piece_id=JEX52J3QTGUWMY3FO6BD5IJV5UPUUCKVV4MYZAOAU3BWMKFSN3KQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.758530774 (600h24m35.629613819s) [noise=true , address=145.79.227.48:4303 , piece_id=IO7FUTELAP2L3VVZEQN6N6QG42WMAOS7H5JEPFAST5YEKMKHYLXA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.796785737 (600h24m35.667868772s) [noise=true , address=145.79.227.3:4150 , piece_id=BUBUCPH3LTJO2CR3IVIXEOJOVS422KQT6L2Z22OBZP6DFPCK5KAA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.029561278 (600h24m35.900644313s) [noise=true , address=77.90.31.10:4517 , piece_id=5Y32J2NPA664L4ZGRDDYYX66FPWND4DNZ5OZXAADJ3LOXK3IA5BA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.070210490 (600h24m35.941293536s) [noise=true , address=45.91.116.13:4153 , piece_id=AQGHSCVDMOODBX225O5SSOLNLJQYPAY2X7EJ7G5W6RP42ZJ4DGCQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.082591286 (600h24m35.953674331s) [noise=true , address=45.91.116.11:4120 , piece_id=ZTXP5ESUQMWFEWTQUX2SMER7W2NJG5IISRWFAM3IKAKQFKJOGWSQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.082610784 (600h24m35.953693819s) [seg_pos={0 0}, upload=526231]: Segment reached optimal threshold of 13 pieces. Dec 30 15:53:36.082858675 (600h24m35.953941711s) [noise=true , address=77.90.31.14:4664 , piece_id=YLXNEMZBXUYTFQ4P6ZFHPA6BIUH4TZ2OBXEQAJWYSCUX333L777A, seg_pos={0 0}, upload=526231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 30 15:53:36.082868053 (600h24m35.953951088s) [noise=true , address=77.90.31.14:4664 , piece_id=YLXNEMZBXUYTFQ4P6ZFHPA6BIUH4TZ2OBXEQAJWYSCUX333L777A, seg_pos={0 0}, upload=526231]: Long tail cancellation due to optimal threshold Dec 30 15:53:36.082953826 (600h24m35.954036871s) [noise=true , address=45.91.116.12:4227 , piece_id=5GQIOQC4VDIF3Z6J4TUPZG3XUC3AACZ53CPYB4JIBQD7F5BK2IFA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 30 15:53:36.082969977 (600h24m35.954053012s) [noise=true , address=45.91.116.12:4227 , piece_id=5GQIOQC4VDIF3Z6J4TUPZG3XUC3AACZ53CPYB4JIBQD7F5BK2IFA, seg_pos={0 0}, upload=526231]: Long tail cancellation due to optimal threshold Dec 30 15:53:36.083266191 (600h24m35.954349226s) [noise=true , address=45.91.116.8:4047 , piece_id=TLKGZQ2VDRAGBZCULZ35ZR6GEJXQEE2HZ35Q52XBEO5C4FTRFICA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: read tcp 188.166.225.170:42816->45.91.116.8:4047: use of closed network connection; context canceled Dec 30 15:53:36.083274006 (600h24m35.954357041s) [noise=true , address=45.91.116.8:4047 , piece_id=TLKGZQ2VDRAGBZCULZ35ZR6GEJXQEE2HZ35Q52XBEO5C4FTRFICA, seg_pos={0 0}, upload=526231]: Long tail cancellation due to optimal threshold Dec 30 15:53:36.083296048 (600h24m35.954379083s) [seg_pos={0 0}, upload=526231]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 15:53:36.083385679 (600h24m35.954468714s) [upload=526231]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 15:53:36.083392341 (600h24m35.954475376s) [upload=526231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 15:53:36.379743600 (600h24m36.250826635s) [upload=526231]: Done uploading segments... Dec 30 15:53:36.379758638 (600h24m36.250841673s) [upload=526231]: Upload finished. err: <nil> Duration: 27.711674466s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 44 out of 60 is failed (threshold=6) [XXXX.XX.X.X.X.XXXXXX.XXXX...XX.X.XXXXXXXXX.XXXXXXX...XXXXX.X] duration: 5.001315381s trace: https://jaeger.dev.storj.io/trace/3605fe9c3b54da5a FAIL (2025-12-30 15:57:41 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001397277s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 43 out of 60 is failed (threshold=6) [.XXXX.XX.X.X.X.XXXXXX.XXXX...XX.X.XXXXXXXXX.XXXXXXX...XXXXX.] duration: 3.639622486s trace: https://jaeger.dev.storj.io/trace/51613ef943cde7f1 FAIL (2025-12-30 15:57:22 UTC) orbiter warning: context deadline exceeded Duration: 3.639643617s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 11.391115549s trace: https://jaeger.dev.storj.io/trace/547648a2cfce50e7 FAIL (2025-12-30 15:57:20 UTC) orbiter warning: context deadline exceeded Duration: 11.3911426s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X............................X..XX..X..................X..] duration: 15.519909958s trace: https://jaeger.dev.storj.io/trace/2ead4e08bdf51a6a FAIL (2025-12-30 15:57:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 43:4696 , piece_id=3WM5R3KUHXN4WVDOKBAOFX7RAI6ZUTV7UKHUOYAKHUYPM4M24N2A, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.239224572 (600h25m38.110307617s) [noise=true , address=45.91.116.12:4241 , piece_id=FYZGK7TFHSQ5BDLEMJIOGXWVWSZ45NRVSYQ4DT6DHBANOCBOZZWA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.239811830 (600h25m38.110894865s) [noise=true , address=145.79.227.43:4189 , piece_id=DYHLPLLBNG6B3FXERO33ZDDSHFUHQIIYGQZJFE2UMJXMLUBYC3KQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.241164706 (600h25m38.112247741s) [noise=true , address=77.90.31.12:4604 , piece_id=OKY6LAK735LFQ4MDC5K6LJPFQITQSGNOZUGNU6US25RELWLK33BQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.244508833 (600h25m38.115591878s) [noise=true , address=145.79.227.49:4350 , piece_id=MA66IA2WTN55SFQDQ4LF7H5LNVUK3IMIHHRUH4TXWHWZPTQDBDCA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.244943361 (600h25m38.116026396s) [noise=true , address=77.90.31.17:4800 , piece_id=K3HNTJHOSIIUBSKISKWG5YRGLKNKHDOWHITXEDDLKPATPIC4IKTQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.250127493 (600h25m38.121210528s) [noise=true , address=145.79.227.35:4516 , piece_id=RDI4UFYB6YC22A5QOGDTLRGCG63TSV6XJFKGLKGSJBC5CH3KCQMQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.250776008 (600h25m38.121859053s) [noise=true , address=145.79.227.39:4033 , piece_id=BLTCPLYP5MIQO5Y7WWPT5QIDV6QM3YWAFMLOEWFOVWFLZOO67QSA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.253191467 (600h25m38.124274512s) [noise=true , address=145.79.227.44:4196 , piece_id=ZUDLORSKFXXIGHZ3YNIE3T7JF7OG55NZKPTSFOUVHAISGF7V7J5Q, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.254554973 (600h25m38.125638018s) [noise=true , address=45.91.116.5:4314 , piece_id=XWWPTNVPYYZ6VL26MACQEJTB2FYN6523YJGZK4I5B4OFRJNUMY3A, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.255815333 (600h25m38.126898378s) [noise=true , address=145.79.227.40:4077 , piece_id=AWXM3BGQVCAARS5F5UISWWIFAVWLWV2KML4FAP6BQ6QNXJTG77NA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.259122731 (600h25m38.130205776s) [noise=true , address=64.49.13.169:4713 , piece_id=KI4XABHC36ADSOANW3HVM3TCWZ4ZXXXNP7MWJVESDKD5WO26DJHQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.265517899 (600h25m38.136600944s) [noise=true , address=77.90.31.19:4854 , piece_id=4W27FHSBO3L2TSAZL2JFCTUPHYV7RI5XDQ4SJWOCRCD5V4K4IK2Q, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.266738002 (600h25m38.137821047s) [noise=true , address=145.79.227.33:4877 , piece_id=GPUOFM2E3KZGDCEZQYPKADLJFDML5DS7OTWFW7AFB5DYAHBVBQ6Q, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.267328907 (600h25m38.138411952s) [noise=true , address=77.90.31.21:4911 , piece_id=B25VZGBKAGIB4ZN6OICFQFZTRQWZIIYDBFUWVNAW4QW26AVYSRPA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:39.692439689 (600h25m39.563522734s) [upload=526249]: Next returned nil segment Dec 30 15:54:39.692451583 (600h25m39.563534618s) [upload=526249]: Waiting for error group managing segments... Dec 30 15:54:40.135912060 (600h25m40.006995095s) [noise=true , address=145.79.227.39:4033 , piece_id=BLTCPLYP5MIQO5Y7WWPT5QIDV6QM3YWAFMLOEWFOVWFLZOO67QSA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.285764442 (600h25m40.156847487s) [noise=true , address=5.231.73.3:4032 , piece_id=7MBVEHMVMT6XTZJZHHWA65RMFNIB7EH4HU63WTLBPPB6CILGMYAA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.400071651 (600h25m40.271154696s) [noise=true , address=77.90.31.12:4604 , piece_id=OKY6LAK735LFQ4MDC5K6LJPFQITQSGNOZUGNU6US25RELWLK33BQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.423162570 (600h25m40.294245605s) [noise=true , address=45.91.116.12:4241 , piece_id=FYZGK7TFHSQ5BDLEMJIOGXWVWSZ45NRVSYQ4DT6DHBANOCBOZZWA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.428962032 (600h25m40.300045067s) [noise=true , address=77.90.31.17:4800 , piece_id=K3HNTJHOSIIUBSKISKWG5YRGLKNKHDOWHITXEDDLKPATPIC4IKTQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.543712124 (600h25m40.414795169s) [noise=true , address=45.91.116.5:4314 , piece_id=XWWPTNVPYYZ6VL26MACQEJTB2FYN6523YJGZK4I5B4OFRJNUMY3A, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.993222051 (600h25m40.864305086s) [noise=true , address=145.79.227.44:4196 , piece_id=ZUDLORSKFXXIGHZ3YNIE3T7JF7OG55NZKPTSFOUVHAISGF7V7J5Q, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.006811979 (600h25m40.877895024s) [noise=true , address=193.149.164.143:4696 , piece_id=3WM5R3KUHXN4WVDOKBAOFX7RAI6ZUTV7UKHUOYAKHUYPM4M24N2A, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.024553631 (600h25m40.895636666s) [noise=true , address=145.79.227.40:4077 , piece_id=AWXM3BGQVCAARS5F5UISWWIFAVWLWV2KML4FAP6BQ6QNXJTG77NA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.054002647 (600h25m40.925085692s) [noise=true , address=145.79.227.49:4350 , piece_id=MA66IA2WTN55SFQDQ4LF7H5LNVUK3IMIHHRUH4TXWHWZPTQDBDCA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.074754852 (600h25m40.945837897s) [noise=true , address=77.90.31.21:4911 , piece_id=B25VZGBKAGIB4ZN6OICFQFZTRQWZIIYDBFUWVNAW4QW26AVYSRPA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.076478905 (600h25m40.94756195s) [noise=true , address=145.79.227.35:4516 , piece_id=RDI4UFYB6YC22A5QOGDTLRGCG63TSV6XJFKGLKGSJBC5CH3KCQMQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.091294145 (600h25m40.96237718s) [noise=true , address=145.79.227.30:4789 , piece_id=5JZFEA3P6EICC2MGUYOQWY4OWOVHXWZLBJBBSVUAMPLV2EHZOZLA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.138722376 (600h25m41.009805421s) [noise=true , address=145.79.227.49:4359 , piece_id=MCTYZULSW2Z2A6HXPREG3GOEYF5QLBPKEB7DDGT32FNTWHC646CQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.138745481 (600h25m41.009828516s) [seg_pos={0 0}, upload=526249]: Segment reached optimal threshold of 14 pieces. Dec 30 15:54:41.138965780 (600h25m41.010048815s) [noise=true , address=145.79.227.33:4877 , piece_id=GPUOFM2E3KZGDCEZQYPKADLJFDML5DS7OTWFW7AFB5DYAHBVBQ6Q, seg_pos={0 0}, upload=526249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17TuauVrjKmehRpXqmK7bemU9V33GDZu9RGBs6fanXvJkV1gdH): protocol: expected piece hash; context canceled; EOF Dec 30 15:54:41.138974496 (600h25m41.010057531s) [noise=true , address=145.79.227.33:4877 , piece_id=GPUOFM2E3KZGDCEZQYPKADLJFDML5DS7OTWFW7AFB5DYAHBVBQ6Q, seg_pos={0 0}, upload=526249]: Long tail cancellation due to optimal threshold Dec 30 15:54:41.139084886 (600h25m41.010167921s) [noise=true , address=77.90.31.19:4854 , piece_id=4W27FHSBO3L2TSAZL2JFCTUPHYV7RI5XDQ4SJWOCRCD5V4K4IK2Q, seg_pos={0 0}, upload=526249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16HwqYhJnhseWEhzzv4QreZ953rB5cKxqFX8swEqaqiRXCRsAE): protocol: write tcp 188.166.225.170:48144->77.90.31.19:4854: use of closed network connection; write tcp 188.166.225.170:48144->77.90.31.19:4854: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48144->77.90.31.19:4854: use of closed network connection Dec 30 15:54:41.139093533 (600h25m41.010176568s) [noise=true , address=77.90.31.19:4854 , piece_id=4W27FHSBO3L2TSAZL2JFCTUPHYV7RI5XDQ4SJWOCRCD5V4K4IK2Q, seg_pos={0 0}, upload=526249]: Long tail cancellation due to optimal threshold Dec 30 15:54:41.139129352 (600h25m41.010212397s) [noise=true , address=145.79.227.43:4189 , piece_id=DYHLPLLBNG6B3FXERO33ZDDSHFUHQIIYGQZJFE2UMJXMLUBYC3KQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7): protocol: expected piece hash; context canceled; EOF Dec 30 15:54:41.139138429 (600h25m41.010221484s) [noise=true , address=145.79.227.43:4189 , piece_id=DYHLPLLBNG6B3FXERO33ZDDSHFUHQIIYGQZJFE2UMJXMLUBYC3KQ, seg_pos={0 0}, upload=526249]: Long tail cancellation due to optimal threshold Dec 30 15:54:41.139238890 (600h25m41.010321925s) [noise=true , address=64.49.13.169:4713 , piece_id=KI4XABHC36ADSOANW3HVM3TCWZ4ZXXXNP7MWJVESDKD5WO26DJHQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:35558->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:35558->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35558->64.49.13.169:4713: use of closed network connection Dec 30 15:54:41.139244420 (600h25m41.010327455s) [noise=true , address=64.49.13.169:4713 , piece_id=KI4XABHC36ADSOANW3HVM3TCWZ4ZXXXNP7MWJVESDKD5WO26DJHQ, seg_pos={0 0}, upload=526249]: Long tail cancellation due to optimal threshold Dec 30 15:54:41.139323681 (600h25m41.010406716s) [seg_pos={0 0}, upload=526249]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 15:54:41.139392121 (600h25m41.010475166s) [upload=526249]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 15:54:41.139396971 (600h25m41.010480006s) [upload=526249]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 15:54:41.438011559 (600h25m41.309094604s) [upload=526249]: Done uploading segments... Dec 30 15:54:41.438036206 (600h25m41.309119251s) [upload=526249]: Upload finished. err: <nil> Duration: 15.519934464s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [....X.X..........XX.......XX.XX..X....X.......X......X.X....] duration: 5.956714898s trace: https://jaeger.dev.storj.io/trace/54fff38414d25b0f FAIL (2025-12-30 15:57:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: AQGHSCVDMOODBX225O5SSOLNLJQYPAY2X7EJ7G5W6RP42ZJ4DGCQ) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: ZTXP5ESUQMWFEWTQUX2SMER7W2NJG5IISRWFAM3IKAKQFKJOGWSQ) context deadline exceeded ======LOGS====== Dec 30 15:53:33.440030169 (600h24m33.311113204s) [upload=526231]: Starting upload Dec 30 15:53:33.440113437 (600h24m33.311196492s) [upload=526231]: Done starting upload Dec 30 15:53:33.440168081 (600h24m33.311251116s) [upload=526231]: Uploading segments... Dec 30 15:53:33.440357231 (600h24m33.311440276s) [upload=526231]: Got next segment. Inline: false Dec 30 15:53:33.440411114 (600h24m33.311494159s) [upload=526231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 15:53:33.440416885 (600h24m33.31149992s) [upload=526231]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 15:53:33.684984430 (600h24m33.556067505s) [seg_pos={0 0}, upload=526231]: Begin upload segment... Dec 30 15:53:33.685033994 (600h24m33.556117029s) [seg_pos={0 0}, upload=526231]: Done begin upload segment. Dec 30 15:53:33.685081235 (600h24m33.55616427s) [noise=true , address=145.79.227.36:4519 , piece_id=5JAZZV45ZTNDSTJZIX2EI7RK7MCBXBQUV2PLSF2HBCRFCIJEA6OQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.685200161 (600h24m33.556283196s) [noise=true , address=145.79.227.23:4545 , piece_id=NHTKPJFEFGAJOEXUNJJ7TU6N27JJP3O5TFSMWF5I2SMGEYXWEAKQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.685709841 (600h24m33.556792887s) [noise=true , address=77.90.31.14:4664 , piece_id=YLXNEMZBXUYTFQ4P6ZFHPA6BIUH4TZ2OBXEQAJWYSCUX333L777A, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686084574 (600h24m33.557167619s) [noise=true , address=45.91.116.12:4227 , piece_id=5GQIOQC4VDIF3Z6J4TUPZG3XUC3AACZ53CPYB4JIBQD7F5BK2IFA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686094794 (600h24m33.55717783s) [noise=true , address=77.90.31.10:4517 , piece_id=5Y32J2NPA664L4ZGRDDYYX66FPWND4DNZ5OZXAADJ3LOXK3IA5BA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686429431 (600h24m33.557512476s) [noise=true , address=45.91.116.13:4153 , piece_id=AQGHSCVDMOODBX225O5SSOLNLJQYPAY2X7EJ7G5W6RP42ZJ4DGCQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686489927 (600h24m33.557572962s) [noise=true , address=45.91.116.8:4047 , piece_id=TLKGZQ2VDRAGBZCULZ35ZR6GEJXQEE2HZ35Q52XBEO5C4FTRFICA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686910547 (600h24m33.557993582s) [noise=true , address=145.79.227.37:4567 , piece_id=WSHVGPU7L73OET2M6WT7VPHGVRQV6ZCJRPAW7AEXBUQDFWQBNAUA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686946536 (600h24m33.558029581s) [noise=true , address=145.79.227.14:4838 , piece_id=YB4VPIAQTBJ6G5XFXDVJ3OL6YFAWMEBHHS63SEDX5U4HYKXT3KJQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.687266755 (600h24m33.55834979s) [noise=true , address=145.79.227.6:4257 , piece_id=53MG7UDQR3MJYNMQXY3IOPPKOQEOI3KFUTGHQUEHOKCNG5NTAWXA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.687625419 (600h24m33.558708464s) [noise=true , address=45.91.116.11:4120 , piece_id=ZTXP5ESUQMWFEWTQUX2SMER7W2NJG5IISRWFAM3IKAKQFKJOGWSQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.688010913 (600h24m33.559093948s) [noise=true , address=145.79.227.19:4077 , piece_id=JEX52J3QTGUWMY3FO6BD5IJV5UPUUCKVV4MYZAOAU3BWMKFSN3KQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.688360248 (600h24m33.559443283s) [noise=true , address=145.79.227.16:4910 , piece_id=UHDFUXEQCMN73AUB7WEJBG5HY2G4E25MQGSBNK6AKHMQMDZCJYRA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.688673685 (600h24m33.55975672s) [noise=true , address=145.79.227.48:4303 , piece_id=IO7FUTELAP2L3VVZEQN6N6QG42WMAOS7H5JEPFAST5YEKMKHYLXA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.689057035 (600h24m33.56014007s) [noise=true , address=145.79.227.2:4114 , piece_id=3MJV2XXBY776ZUIO2KU4MEP37SP6P2NYFO255DCI6IFUZWUAGOWQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.690890025 (600h24m33.56197308s) [noise=true , address=145.79.227.3:4150 , piece_id=BUBUCPH3LTJO2CR3IVIXEOJOVS422KQT6L2Z22OBZP6DFPCK5KAA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:35.401137462 (600h24m35.272220507s) [upload=526231]: Next returned nil segment Dec 30 15:53:35.401154925 (600h24m35.27223797s) [upload=526231]: Waiting for error group managing segments... Dec 30 15:53:35.673442202 (600h24m35.544525237s) [noise=true , address=145.79.227.23:4545 , piece_id=NHTKPJFEFGAJOEXUNJJ7TU6N27JJP3O5TFSMWF5I2SMGEYXWEAKQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.674880822 (600h24m35.545963857s) [noise=true , address=145.79.227.2:4114 , piece_id=3MJV2XXBY776ZUIO2KU4MEP37SP6P2NYFO255DCI6IFUZWUAGOWQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.688801769 (600h24m35.559884814s) [noise=true , address=145.79.227.36:4519 , piece_id=5JAZZV45ZTNDSTJZIX2EI7RK7MCBXBQUV2PLSF2HBCRFCIJEA6OQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.689704678 (600h24m35.560787713s) [noise=true , address=145.79.227.6:4257 , piece_id=53MG7UDQR3MJYNMQXY3IOPPKOQEOI3KFUTGHQUEHOKCNG5NTAWXA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.690785346 (600h24m35.561868381s) [noise=true , address=145.79.227.14:4838 , piece_id=YB4VPIAQTBJ6G5XFXDVJ3OL6YFAWMEBHHS63SEDX5U4HYKXT3KJQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.692882529 (600h24m35.563965564s) [noise=true , address=145.79.227.37:4567 , piece_id=WSHVGPU7L73OET2M6WT7VPHGVRQV6ZCJRPAW7AEXBUQDFWQBNAUA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.696575461 (600h24m35.567658506s) [noise=true , address=145.79.227.16:4910 , piece_id=UHDFUXEQCMN73AUB7WEJBG5HY2G4E25MQGSBNK6AKHMQMDZCJYRA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.697864125 (600h24m35.56894716s) [noise=true , address=145.79.227.19:4077 , piece_id=JEX52J3QTGUWMY3FO6BD5IJV5UPUUCKVV4MYZAOAU3BWMKFSN3KQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.758530774 (600h24m35.629613819s) [noise=true , address=145.79.227.48:4303 , piece_id=IO7FUTELAP2L3VVZEQN6N6QG42WMAOS7H5JEPFAST5YEKMKHYLXA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.796785737 (600h24m35.667868772s) [noise=true , address=145.79.227.3:4150 , piece_id=BUBUCPH3LTJO2CR3IVIXEOJOVS422KQT6L2Z22OBZP6DFPCK5KAA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.029561278 (600h24m35.900644313s) [noise=true , address=77.90.31.10:4517 , piece_id=5Y32J2NPA664L4ZGRDDYYX66FPWND4DNZ5OZXAADJ3LOXK3IA5BA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.070210490 (600h24m35.941293536s) [noise=true , address=45.91.116.13:4153 , piece_id=AQGHSCVDMOODBX225O5SSOLNLJQYPAY2X7EJ7G5W6RP42ZJ4DGCQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.082591286 (600h24m35.953674331s) [noise=true , address=45.91.116.11:4120 , piece_id=ZTXP5ESUQMWFEWTQUX2SMER7W2NJG5IISRWFAM3IKAKQFKJOGWSQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.082610784 (600h24m35.953693819s) [seg_pos={0 0}, upload=526231]: Segment reached optimal threshold of 13 pieces. Dec 30 15:53:36.082858675 (600h24m35.953941711s) [noise=true , address=77.90.31.14:4664 , piece_id=YLXNEMZBXUYTFQ4P6ZFHPA6BIUH4TZ2OBXEQAJWYSCUX333L777A, seg_pos={0 0}, upload=526231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 30 15:53:36.082868053 (600h24m35.953951088s) [noise=true , address=77.90.31.14:4664 , piece_id=YLXNEMZBXUYTFQ4P6ZFHPA6BIUH4TZ2OBXEQAJWYSCUX333L777A, seg_pos={0 0}, upload=526231]: Long tail cancellation due to optimal threshold Dec 30 15:53:36.082953826 (600h24m35.954036871s) [noise=true , address=45.91.116.12:4227 , piece_id=5GQIOQC4VDIF3Z6J4TUPZG3XUC3AACZ53CPYB4JIBQD7F5BK2IFA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 30 15:53:36.082969977 (600h24m35.954053012s) [noise=true , address=45.91.116.12:4227 , piece_id=5GQIOQC4VDIF3Z6J4TUPZG3XUC3AACZ53CPYB4JIBQD7F5BK2IFA, seg_pos={0 0}, upload=526231]: Long tail cancellation due to optimal threshold Dec 30 15:53:36.083266191 (600h24m35.954349226s) [noise=true , address=45.91.116.8:4047 , piece_id=TLKGZQ2VDRAGBZCULZ35ZR6GEJXQEE2HZ35Q52XBEO5C4FTRFICA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: read tcp 188.166.225.170:42816->45.91.116.8:4047: use of closed network connection; context canceled Dec 30 15:53:36.083274006 (600h24m35.954357041s) [noise=true , address=45.91.116.8:4047 , piece_id=TLKGZQ2VDRAGBZCULZ35ZR6GEJXQEE2HZ35Q52XBEO5C4FTRFICA, seg_pos={0 0}, upload=526231]: Long tail cancellation due to optimal threshold Dec 30 15:53:36.083296048 (600h24m35.954379083s) [seg_pos={0 0}, upload=526231]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 15:53:36.083385679 (600h24m35.954468714s) [upload=526231]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 15:53:36.083392341 (600h24m35.954475376s) [upload=526231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 15:53:36.379743600 (600h24m36.250826635s) [upload=526231]: Done uploading segments... Dec 30 15:53:36.379758638 (600h24m36.250841673s) [upload=526231]: Upload finished. err: <nil> Duration: 5.956746007s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 12.379093807s trace: https://jaeger.dev.storj.io/trace/547648a2cfce50e7 FAIL (2025-12-30 15:56:46 UTC) orbiter warning: context deadline exceeded Duration: 12.379120537s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 43 out of 60 is failed (threshold=6) [..XXXX.XX.X.X.X.XXXXXX.XXXX...XX.X.XXXXXXXXX.XXXXXXX...XXXXX] duration: 5.001177686s trace: https://jaeger.dev.storj.io/trace/51613ef943cde7f1 FAIL (2025-12-30 15:56:30 UTC) orbiter warning: context deadline exceeded Duration: 5.001200029s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 11.347210564s trace: https://jaeger.dev.storj.io/trace/547648a2cfce50e7 FAIL (2025-12-30 15:56:17 UTC) orbiter warning: context deadline exceeded Duration: 11.347285768s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.....X.X..........XX.......XX.XX..X....X.......X......X.X...] duration: 5.612959131s trace: https://jaeger.dev.storj.io/trace/54fff38414d25b0f FAIL (2025-12-30 15:56:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: AQGHSCVDMOODBX225O5SSOLNLJQYPAY2X7EJ7G5W6RP42ZJ4DGCQ) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: ZTXP5ESUQMWFEWTQUX2SMER7W2NJG5IISRWFAM3IKAKQFKJOGWSQ) context deadline exceeded ======LOGS====== Dec 30 15:53:33.440030169 (600h24m33.311113204s) [upload=526231]: Starting upload Dec 30 15:53:33.440113437 (600h24m33.311196492s) [upload=526231]: Done starting upload Dec 30 15:53:33.440168081 (600h24m33.311251116s) [upload=526231]: Uploading segments... Dec 30 15:53:33.440357231 (600h24m33.311440276s) [upload=526231]: Got next segment. Inline: false Dec 30 15:53:33.440411114 (600h24m33.311494159s) [upload=526231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 15:53:33.440416885 (600h24m33.31149992s) [upload=526231]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 15:53:33.684984430 (600h24m33.556067505s) [seg_pos={0 0}, upload=526231]: Begin upload segment... Dec 30 15:53:33.685033994 (600h24m33.556117029s) [seg_pos={0 0}, upload=526231]: Done begin upload segment. Dec 30 15:53:33.685081235 (600h24m33.55616427s) [noise=true , address=145.79.227.36:4519 , piece_id=5JAZZV45ZTNDSTJZIX2EI7RK7MCBXBQUV2PLSF2HBCRFCIJEA6OQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.685200161 (600h24m33.556283196s) [noise=true , address=145.79.227.23:4545 , piece_id=NHTKPJFEFGAJOEXUNJJ7TU6N27JJP3O5TFSMWF5I2SMGEYXWEAKQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.685709841 (600h24m33.556792887s) [noise=true , address=77.90.31.14:4664 , piece_id=YLXNEMZBXUYTFQ4P6ZFHPA6BIUH4TZ2OBXEQAJWYSCUX333L777A, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686084574 (600h24m33.557167619s) [noise=true , address=45.91.116.12:4227 , piece_id=5GQIOQC4VDIF3Z6J4TUPZG3XUC3AACZ53CPYB4JIBQD7F5BK2IFA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686094794 (600h24m33.55717783s) [noise=true , address=77.90.31.10:4517 , piece_id=5Y32J2NPA664L4ZGRDDYYX66FPWND4DNZ5OZXAADJ3LOXK3IA5BA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686429431 (600h24m33.557512476s) [noise=true , address=45.91.116.13:4153 , piece_id=AQGHSCVDMOODBX225O5SSOLNLJQYPAY2X7EJ7G5W6RP42ZJ4DGCQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686489927 (600h24m33.557572962s) [noise=true , address=45.91.116.8:4047 , piece_id=TLKGZQ2VDRAGBZCULZ35ZR6GEJXQEE2HZ35Q52XBEO5C4FTRFICA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686910547 (600h24m33.557993582s) [noise=true , address=145.79.227.37:4567 , piece_id=WSHVGPU7L73OET2M6WT7VPHGVRQV6ZCJRPAW7AEXBUQDFWQBNAUA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686946536 (600h24m33.558029581s) [noise=true , address=145.79.227.14:4838 , piece_id=YB4VPIAQTBJ6G5XFXDVJ3OL6YFAWMEBHHS63SEDX5U4HYKXT3KJQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.687266755 (600h24m33.55834979s) [noise=true , address=145.79.227.6:4257 , piece_id=53MG7UDQR3MJYNMQXY3IOPPKOQEOI3KFUTGHQUEHOKCNG5NTAWXA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.687625419 (600h24m33.558708464s) [noise=true , address=45.91.116.11:4120 , piece_id=ZTXP5ESUQMWFEWTQUX2SMER7W2NJG5IISRWFAM3IKAKQFKJOGWSQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.688010913 (600h24m33.559093948s) [noise=true , address=145.79.227.19:4077 , piece_id=JEX52J3QTGUWMY3FO6BD5IJV5UPUUCKVV4MYZAOAU3BWMKFSN3KQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.688360248 (600h24m33.559443283s) [noise=true , address=145.79.227.16:4910 , piece_id=UHDFUXEQCMN73AUB7WEJBG5HY2G4E25MQGSBNK6AKHMQMDZCJYRA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.688673685 (600h24m33.55975672s) [noise=true , address=145.79.227.48:4303 , piece_id=IO7FUTELAP2L3VVZEQN6N6QG42WMAOS7H5JEPFAST5YEKMKHYLXA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.689057035 (600h24m33.56014007s) [noise=true , address=145.79.227.2:4114 , piece_id=3MJV2XXBY776ZUIO2KU4MEP37SP6P2NYFO255DCI6IFUZWUAGOWQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.690890025 (600h24m33.56197308s) [noise=true , address=145.79.227.3:4150 , piece_id=BUBUCPH3LTJO2CR3IVIXEOJOVS422KQT6L2Z22OBZP6DFPCK5KAA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:35.401137462 (600h24m35.272220507s) [upload=526231]: Next returned nil segment Dec 30 15:53:35.401154925 (600h24m35.27223797s) [upload=526231]: Waiting for error group managing segments... Dec 30 15:53:35.673442202 (600h24m35.544525237s) [noise=true , address=145.79.227.23:4545 , piece_id=NHTKPJFEFGAJOEXUNJJ7TU6N27JJP3O5TFSMWF5I2SMGEYXWEAKQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.674880822 (600h24m35.545963857s) [noise=true , address=145.79.227.2:4114 , piece_id=3MJV2XXBY776ZUIO2KU4MEP37SP6P2NYFO255DCI6IFUZWUAGOWQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.688801769 (600h24m35.559884814s) [noise=true , address=145.79.227.36:4519 , piece_id=5JAZZV45ZTNDSTJZIX2EI7RK7MCBXBQUV2PLSF2HBCRFCIJEA6OQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.689704678 (600h24m35.560787713s) [noise=true , address=145.79.227.6:4257 , piece_id=53MG7UDQR3MJYNMQXY3IOPPKOQEOI3KFUTGHQUEHOKCNG5NTAWXA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.690785346 (600h24m35.561868381s) [noise=true , address=145.79.227.14:4838 , piece_id=YB4VPIAQTBJ6G5XFXDVJ3OL6YFAWMEBHHS63SEDX5U4HYKXT3KJQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.692882529 (600h24m35.563965564s) [noise=true , address=145.79.227.37:4567 , piece_id=WSHVGPU7L73OET2M6WT7VPHGVRQV6ZCJRPAW7AEXBUQDFWQBNAUA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.696575461 (600h24m35.567658506s) [noise=true , address=145.79.227.16:4910 , piece_id=UHDFUXEQCMN73AUB7WEJBG5HY2G4E25MQGSBNK6AKHMQMDZCJYRA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.697864125 (600h24m35.56894716s) [noise=true , address=145.79.227.19:4077 , piece_id=JEX52J3QTGUWMY3FO6BD5IJV5UPUUCKVV4MYZAOAU3BWMKFSN3KQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.758530774 (600h24m35.629613819s) [noise=true , address=145.79.227.48:4303 , piece_id=IO7FUTELAP2L3VVZEQN6N6QG42WMAOS7H5JEPFAST5YEKMKHYLXA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.796785737 (600h24m35.667868772s) [noise=true , address=145.79.227.3:4150 , piece_id=BUBUCPH3LTJO2CR3IVIXEOJOVS422KQT6L2Z22OBZP6DFPCK5KAA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.029561278 (600h24m35.900644313s) [noise=true , address=77.90.31.10:4517 , piece_id=5Y32J2NPA664L4ZGRDDYYX66FPWND4DNZ5OZXAADJ3LOXK3IA5BA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.070210490 (600h24m35.941293536s) [noise=true , address=45.91.116.13:4153 , piece_id=AQGHSCVDMOODBX225O5SSOLNLJQYPAY2X7EJ7G5W6RP42ZJ4DGCQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.082591286 (600h24m35.953674331s) [noise=true , address=45.91.116.11:4120 , piece_id=ZTXP5ESUQMWFEWTQUX2SMER7W2NJG5IISRWFAM3IKAKQFKJOGWSQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.082610784 (600h24m35.953693819s) [seg_pos={0 0}, upload=526231]: Segment reached optimal threshold of 13 pieces. Dec 30 15:53:36.082858675 (600h24m35.953941711s) [noise=true , address=77.90.31.14:4664 , piece_id=YLXNEMZBXUYTFQ4P6ZFHPA6BIUH4TZ2OBXEQAJWYSCUX333L777A, seg_pos={0 0}, upload=526231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 30 15:53:36.082868053 (600h24m35.953951088s) [noise=true , address=77.90.31.14:4664 , piece_id=YLXNEMZBXUYTFQ4P6ZFHPA6BIUH4TZ2OBXEQAJWYSCUX333L777A, seg_pos={0 0}, upload=526231]: Long tail cancellation due to optimal threshold Dec 30 15:53:36.082953826 (600h24m35.954036871s) [noise=true , address=45.91.116.12:4227 , piece_id=5GQIOQC4VDIF3Z6J4TUPZG3XUC3AACZ53CPYB4JIBQD7F5BK2IFA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 30 15:53:36.082969977 (600h24m35.954053012s) [noise=true , address=45.91.116.12:4227 , piece_id=5GQIOQC4VDIF3Z6J4TUPZG3XUC3AACZ53CPYB4JIBQD7F5BK2IFA, seg_pos={0 0}, upload=526231]: Long tail cancellation due to optimal threshold Dec 30 15:53:36.083266191 (600h24m35.954349226s) [noise=true , address=45.91.116.8:4047 , piece_id=TLKGZQ2VDRAGBZCULZ35ZR6GEJXQEE2HZ35Q52XBEO5C4FTRFICA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: read tcp 188.166.225.170:42816->45.91.116.8:4047: use of closed network connection; context canceled Dec 30 15:53:36.083274006 (600h24m35.954357041s) [noise=true , address=45.91.116.8:4047 , piece_id=TLKGZQ2VDRAGBZCULZ35ZR6GEJXQEE2HZ35Q52XBEO5C4FTRFICA, seg_pos={0 0}, upload=526231]: Long tail cancellation due to optimal threshold Dec 30 15:53:36.083296048 (600h24m35.954379083s) [seg_pos={0 0}, upload=526231]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 15:53:36.083385679 (600h24m35.954468714s) [upload=526231]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 15:53:36.083392341 (600h24m35.954475376s) [upload=526231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 15:53:36.379743600 (600h24m36.250826635s) [upload=526231]: Done uploading segments... Dec 30 15:53:36.379758638 (600h24m36.250841673s) [upload=526231]: Upload finished. err: <nil> Duration: 5.6129892s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [...X............................X..XX..X..................X.] duration: 26.475847194s trace: https://jaeger.dev.storj.io/trace/2ead4e08bdf51a6a FAIL (2025-12-30 15:56:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 43:4696 , piece_id=3WM5R3KUHXN4WVDOKBAOFX7RAI6ZUTV7UKHUOYAKHUYPM4M24N2A, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.239224572 (600h25m38.110307617s) [noise=true , address=45.91.116.12:4241 , piece_id=FYZGK7TFHSQ5BDLEMJIOGXWVWSZ45NRVSYQ4DT6DHBANOCBOZZWA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.239811830 (600h25m38.110894865s) [noise=true , address=145.79.227.43:4189 , piece_id=DYHLPLLBNG6B3FXERO33ZDDSHFUHQIIYGQZJFE2UMJXMLUBYC3KQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.241164706 (600h25m38.112247741s) [noise=true , address=77.90.31.12:4604 , piece_id=OKY6LAK735LFQ4MDC5K6LJPFQITQSGNOZUGNU6US25RELWLK33BQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.244508833 (600h25m38.115591878s) [noise=true , address=145.79.227.49:4350 , piece_id=MA66IA2WTN55SFQDQ4LF7H5LNVUK3IMIHHRUH4TXWHWZPTQDBDCA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.244943361 (600h25m38.116026396s) [noise=true , address=77.90.31.17:4800 , piece_id=K3HNTJHOSIIUBSKISKWG5YRGLKNKHDOWHITXEDDLKPATPIC4IKTQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.250127493 (600h25m38.121210528s) [noise=true , address=145.79.227.35:4516 , piece_id=RDI4UFYB6YC22A5QOGDTLRGCG63TSV6XJFKGLKGSJBC5CH3KCQMQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.250776008 (600h25m38.121859053s) [noise=true , address=145.79.227.39:4033 , piece_id=BLTCPLYP5MIQO5Y7WWPT5QIDV6QM3YWAFMLOEWFOVWFLZOO67QSA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.253191467 (600h25m38.124274512s) [noise=true , address=145.79.227.44:4196 , piece_id=ZUDLORSKFXXIGHZ3YNIE3T7JF7OG55NZKPTSFOUVHAISGF7V7J5Q, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.254554973 (600h25m38.125638018s) [noise=true , address=45.91.116.5:4314 , piece_id=XWWPTNVPYYZ6VL26MACQEJTB2FYN6523YJGZK4I5B4OFRJNUMY3A, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.255815333 (600h25m38.126898378s) [noise=true , address=145.79.227.40:4077 , piece_id=AWXM3BGQVCAARS5F5UISWWIFAVWLWV2KML4FAP6BQ6QNXJTG77NA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.259122731 (600h25m38.130205776s) [noise=true , address=64.49.13.169:4713 , piece_id=KI4XABHC36ADSOANW3HVM3TCWZ4ZXXXNP7MWJVESDKD5WO26DJHQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.265517899 (600h25m38.136600944s) [noise=true , address=77.90.31.19:4854 , piece_id=4W27FHSBO3L2TSAZL2JFCTUPHYV7RI5XDQ4SJWOCRCD5V4K4IK2Q, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.266738002 (600h25m38.137821047s) [noise=true , address=145.79.227.33:4877 , piece_id=GPUOFM2E3KZGDCEZQYPKADLJFDML5DS7OTWFW7AFB5DYAHBVBQ6Q, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.267328907 (600h25m38.138411952s) [noise=true , address=77.90.31.21:4911 , piece_id=B25VZGBKAGIB4ZN6OICFQFZTRQWZIIYDBFUWVNAW4QW26AVYSRPA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:39.692439689 (600h25m39.563522734s) [upload=526249]: Next returned nil segment Dec 30 15:54:39.692451583 (600h25m39.563534618s) [upload=526249]: Waiting for error group managing segments... Dec 30 15:54:40.135912060 (600h25m40.006995095s) [noise=true , address=145.79.227.39:4033 , piece_id=BLTCPLYP5MIQO5Y7WWPT5QIDV6QM3YWAFMLOEWFOVWFLZOO67QSA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.285764442 (600h25m40.156847487s) [noise=true , address=5.231.73.3:4032 , piece_id=7MBVEHMVMT6XTZJZHHWA65RMFNIB7EH4HU63WTLBPPB6CILGMYAA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.400071651 (600h25m40.271154696s) [noise=true , address=77.90.31.12:4604 , piece_id=OKY6LAK735LFQ4MDC5K6LJPFQITQSGNOZUGNU6US25RELWLK33BQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.423162570 (600h25m40.294245605s) [noise=true , address=45.91.116.12:4241 , piece_id=FYZGK7TFHSQ5BDLEMJIOGXWVWSZ45NRVSYQ4DT6DHBANOCBOZZWA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.428962032 (600h25m40.300045067s) [noise=true , address=77.90.31.17:4800 , piece_id=K3HNTJHOSIIUBSKISKWG5YRGLKNKHDOWHITXEDDLKPATPIC4IKTQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.543712124 (600h25m40.414795169s) [noise=true , address=45.91.116.5:4314 , piece_id=XWWPTNVPYYZ6VL26MACQEJTB2FYN6523YJGZK4I5B4OFRJNUMY3A, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.993222051 (600h25m40.864305086s) [noise=true , address=145.79.227.44:4196 , piece_id=ZUDLORSKFXXIGHZ3YNIE3T7JF7OG55NZKPTSFOUVHAISGF7V7J5Q, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.006811979 (600h25m40.877895024s) [noise=true , address=193.149.164.143:4696 , piece_id=3WM5R3KUHXN4WVDOKBAOFX7RAI6ZUTV7UKHUOYAKHUYPM4M24N2A, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.024553631 (600h25m40.895636666s) [noise=true , address=145.79.227.40:4077 , piece_id=AWXM3BGQVCAARS5F5UISWWIFAVWLWV2KML4FAP6BQ6QNXJTG77NA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.054002647 (600h25m40.925085692s) [noise=true , address=145.79.227.49:4350 , piece_id=MA66IA2WTN55SFQDQ4LF7H5LNVUK3IMIHHRUH4TXWHWZPTQDBDCA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.074754852 (600h25m40.945837897s) [noise=true , address=77.90.31.21:4911 , piece_id=B25VZGBKAGIB4ZN6OICFQFZTRQWZIIYDBFUWVNAW4QW26AVYSRPA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.076478905 (600h25m40.94756195s) [noise=true , address=145.79.227.35:4516 , piece_id=RDI4UFYB6YC22A5QOGDTLRGCG63TSV6XJFKGLKGSJBC5CH3KCQMQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.091294145 (600h25m40.96237718s) [noise=true , address=145.79.227.30:4789 , piece_id=5JZFEA3P6EICC2MGUYOQWY4OWOVHXWZLBJBBSVUAMPLV2EHZOZLA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.138722376 (600h25m41.009805421s) [noise=true , address=145.79.227.49:4359 , piece_id=MCTYZULSW2Z2A6HXPREG3GOEYF5QLBPKEB7DDGT32FNTWHC646CQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.138745481 (600h25m41.009828516s) [seg_pos={0 0}, upload=526249]: Segment reached optimal threshold of 14 pieces. Dec 30 15:54:41.138965780 (600h25m41.010048815s) [noise=true , address=145.79.227.33:4877 , piece_id=GPUOFM2E3KZGDCEZQYPKADLJFDML5DS7OTWFW7AFB5DYAHBVBQ6Q, seg_pos={0 0}, upload=526249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17TuauVrjKmehRpXqmK7bemU9V33GDZu9RGBs6fanXvJkV1gdH): protocol: expected piece hash; context canceled; EOF Dec 30 15:54:41.138974496 (600h25m41.010057531s) [noise=true , address=145.79.227.33:4877 , piece_id=GPUOFM2E3KZGDCEZQYPKADLJFDML5DS7OTWFW7AFB5DYAHBVBQ6Q, seg_pos={0 0}, upload=526249]: Long tail cancellation due to optimal threshold Dec 30 15:54:41.139084886 (600h25m41.010167921s) [noise=true , address=77.90.31.19:4854 , piece_id=4W27FHSBO3L2TSAZL2JFCTUPHYV7RI5XDQ4SJWOCRCD5V4K4IK2Q, seg_pos={0 0}, upload=526249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16HwqYhJnhseWEhzzv4QreZ953rB5cKxqFX8swEqaqiRXCRsAE): protocol: write tcp 188.166.225.170:48144->77.90.31.19:4854: use of closed network connection; write tcp 188.166.225.170:48144->77.90.31.19:4854: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48144->77.90.31.19:4854: use of closed network connection Dec 30 15:54:41.139093533 (600h25m41.010176568s) [noise=true , address=77.90.31.19:4854 , piece_id=4W27FHSBO3L2TSAZL2JFCTUPHYV7RI5XDQ4SJWOCRCD5V4K4IK2Q, seg_pos={0 0}, upload=526249]: Long tail cancellation due to optimal threshold Dec 30 15:54:41.139129352 (600h25m41.010212397s) [noise=true , address=145.79.227.43:4189 , piece_id=DYHLPLLBNG6B3FXERO33ZDDSHFUHQIIYGQZJFE2UMJXMLUBYC3KQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7): protocol: expected piece hash; context canceled; EOF Dec 30 15:54:41.139138429 (600h25m41.010221484s) [noise=true , address=145.79.227.43:4189 , piece_id=DYHLPLLBNG6B3FXERO33ZDDSHFUHQIIYGQZJFE2UMJXMLUBYC3KQ, seg_pos={0 0}, upload=526249]: Long tail cancellation due to optimal threshold Dec 30 15:54:41.139238890 (600h25m41.010321925s) [noise=true , address=64.49.13.169:4713 , piece_id=KI4XABHC36ADSOANW3HVM3TCWZ4ZXXXNP7MWJVESDKD5WO26DJHQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:35558->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:35558->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35558->64.49.13.169:4713: use of closed network connection Dec 30 15:54:41.139244420 (600h25m41.010327455s) [noise=true , address=64.49.13.169:4713 , piece_id=KI4XABHC36ADSOANW3HVM3TCWZ4ZXXXNP7MWJVESDKD5WO26DJHQ, seg_pos={0 0}, upload=526249]: Long tail cancellation due to optimal threshold Dec 30 15:54:41.139323681 (600h25m41.010406716s) [seg_pos={0 0}, upload=526249]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 15:54:41.139392121 (600h25m41.010475166s) [upload=526249]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 15:54:41.139396971 (600h25m41.010480006s) [upload=526249]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 15:54:41.438011559 (600h25m41.309094604s) [upload=526249]: Done uploading segments... Dec 30 15:54:41.438036206 (600h25m41.309119251s) [upload=526249]: Upload finished. err: <nil> Duration: 26.475861562s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 9.759105695s trace: https://jaeger.dev.storj.io/trace/547648a2cfce50e7 FAIL (2025-12-30 15:55:41 UTC) orbiter warning: context deadline exceeded Duration: 9.759133278s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [....X............................X..XX..X..................X] duration: 1m0.003159686s trace: https://jaeger.dev.storj.io/trace/2ead4e08bdf51a6a FAIL (2025-12-30 15:55:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 43:4696 , piece_id=3WM5R3KUHXN4WVDOKBAOFX7RAI6ZUTV7UKHUOYAKHUYPM4M24N2A, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.239224572 (600h25m38.110307617s) [noise=true , address=45.91.116.12:4241 , piece_id=FYZGK7TFHSQ5BDLEMJIOGXWVWSZ45NRVSYQ4DT6DHBANOCBOZZWA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.239811830 (600h25m38.110894865s) [noise=true , address=145.79.227.43:4189 , piece_id=DYHLPLLBNG6B3FXERO33ZDDSHFUHQIIYGQZJFE2UMJXMLUBYC3KQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.241164706 (600h25m38.112247741s) [noise=true , address=77.90.31.12:4604 , piece_id=OKY6LAK735LFQ4MDC5K6LJPFQITQSGNOZUGNU6US25RELWLK33BQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.244508833 (600h25m38.115591878s) [noise=true , address=145.79.227.49:4350 , piece_id=MA66IA2WTN55SFQDQ4LF7H5LNVUK3IMIHHRUH4TXWHWZPTQDBDCA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.244943361 (600h25m38.116026396s) [noise=true , address=77.90.31.17:4800 , piece_id=K3HNTJHOSIIUBSKISKWG5YRGLKNKHDOWHITXEDDLKPATPIC4IKTQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.250127493 (600h25m38.121210528s) [noise=true , address=145.79.227.35:4516 , piece_id=RDI4UFYB6YC22A5QOGDTLRGCG63TSV6XJFKGLKGSJBC5CH3KCQMQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.250776008 (600h25m38.121859053s) [noise=true , address=145.79.227.39:4033 , piece_id=BLTCPLYP5MIQO5Y7WWPT5QIDV6QM3YWAFMLOEWFOVWFLZOO67QSA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.253191467 (600h25m38.124274512s) [noise=true , address=145.79.227.44:4196 , piece_id=ZUDLORSKFXXIGHZ3YNIE3T7JF7OG55NZKPTSFOUVHAISGF7V7J5Q, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.254554973 (600h25m38.125638018s) [noise=true , address=45.91.116.5:4314 , piece_id=XWWPTNVPYYZ6VL26MACQEJTB2FYN6523YJGZK4I5B4OFRJNUMY3A, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.255815333 (600h25m38.126898378s) [noise=true , address=145.79.227.40:4077 , piece_id=AWXM3BGQVCAARS5F5UISWWIFAVWLWV2KML4FAP6BQ6QNXJTG77NA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.259122731 (600h25m38.130205776s) [noise=true , address=64.49.13.169:4713 , piece_id=KI4XABHC36ADSOANW3HVM3TCWZ4ZXXXNP7MWJVESDKD5WO26DJHQ, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.265517899 (600h25m38.136600944s) [noise=true , address=77.90.31.19:4854 , piece_id=4W27FHSBO3L2TSAZL2JFCTUPHYV7RI5XDQ4SJWOCRCD5V4K4IK2Q, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.266738002 (600h25m38.137821047s) [noise=true , address=145.79.227.33:4877 , piece_id=GPUOFM2E3KZGDCEZQYPKADLJFDML5DS7OTWFW7AFB5DYAHBVBQ6Q, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:38.267328907 (600h25m38.138411952s) [noise=true , address=77.90.31.21:4911 , piece_id=B25VZGBKAGIB4ZN6OICFQFZTRQWZIIYDBFUWVNAW4QW26AVYSRPA, seg_pos={0 0}, upload=526249]: Uploading piece... Dec 30 15:54:39.692439689 (600h25m39.563522734s) [upload=526249]: Next returned nil segment Dec 30 15:54:39.692451583 (600h25m39.563534618s) [upload=526249]: Waiting for error group managing segments... Dec 30 15:54:40.135912060 (600h25m40.006995095s) [noise=true , address=145.79.227.39:4033 , piece_id=BLTCPLYP5MIQO5Y7WWPT5QIDV6QM3YWAFMLOEWFOVWFLZOO67QSA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.285764442 (600h25m40.156847487s) [noise=true , address=5.231.73.3:4032 , piece_id=7MBVEHMVMT6XTZJZHHWA65RMFNIB7EH4HU63WTLBPPB6CILGMYAA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.400071651 (600h25m40.271154696s) [noise=true , address=77.90.31.12:4604 , piece_id=OKY6LAK735LFQ4MDC5K6LJPFQITQSGNOZUGNU6US25RELWLK33BQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.423162570 (600h25m40.294245605s) [noise=true , address=45.91.116.12:4241 , piece_id=FYZGK7TFHSQ5BDLEMJIOGXWVWSZ45NRVSYQ4DT6DHBANOCBOZZWA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.428962032 (600h25m40.300045067s) [noise=true , address=77.90.31.17:4800 , piece_id=K3HNTJHOSIIUBSKISKWG5YRGLKNKHDOWHITXEDDLKPATPIC4IKTQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.543712124 (600h25m40.414795169s) [noise=true , address=45.91.116.5:4314 , piece_id=XWWPTNVPYYZ6VL26MACQEJTB2FYN6523YJGZK4I5B4OFRJNUMY3A, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:40.993222051 (600h25m40.864305086s) [noise=true , address=145.79.227.44:4196 , piece_id=ZUDLORSKFXXIGHZ3YNIE3T7JF7OG55NZKPTSFOUVHAISGF7V7J5Q, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.006811979 (600h25m40.877895024s) [noise=true , address=193.149.164.143:4696 , piece_id=3WM5R3KUHXN4WVDOKBAOFX7RAI6ZUTV7UKHUOYAKHUYPM4M24N2A, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.024553631 (600h25m40.895636666s) [noise=true , address=145.79.227.40:4077 , piece_id=AWXM3BGQVCAARS5F5UISWWIFAVWLWV2KML4FAP6BQ6QNXJTG77NA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.054002647 (600h25m40.925085692s) [noise=true , address=145.79.227.49:4350 , piece_id=MA66IA2WTN55SFQDQ4LF7H5LNVUK3IMIHHRUH4TXWHWZPTQDBDCA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.074754852 (600h25m40.945837897s) [noise=true , address=77.90.31.21:4911 , piece_id=B25VZGBKAGIB4ZN6OICFQFZTRQWZIIYDBFUWVNAW4QW26AVYSRPA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.076478905 (600h25m40.94756195s) [noise=true , address=145.79.227.35:4516 , piece_id=RDI4UFYB6YC22A5QOGDTLRGCG63TSV6XJFKGLKGSJBC5CH3KCQMQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.091294145 (600h25m40.96237718s) [noise=true , address=145.79.227.30:4789 , piece_id=5JZFEA3P6EICC2MGUYOQWY4OWOVHXWZLBJBBSVUAMPLV2EHZOZLA, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.138722376 (600h25m41.009805421s) [noise=true , address=145.79.227.49:4359 , piece_id=MCTYZULSW2Z2A6HXPREG3GOEYF5QLBPKEB7DDGT32FNTWHC646CQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: <nil> Dec 30 15:54:41.138745481 (600h25m41.009828516s) [seg_pos={0 0}, upload=526249]: Segment reached optimal threshold of 14 pieces. Dec 30 15:54:41.138965780 (600h25m41.010048815s) [noise=true , address=145.79.227.33:4877 , piece_id=GPUOFM2E3KZGDCEZQYPKADLJFDML5DS7OTWFW7AFB5DYAHBVBQ6Q, seg_pos={0 0}, upload=526249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17TuauVrjKmehRpXqmK7bemU9V33GDZu9RGBs6fanXvJkV1gdH): protocol: expected piece hash; context canceled; EOF Dec 30 15:54:41.138974496 (600h25m41.010057531s) [noise=true , address=145.79.227.33:4877 , piece_id=GPUOFM2E3KZGDCEZQYPKADLJFDML5DS7OTWFW7AFB5DYAHBVBQ6Q, seg_pos={0 0}, upload=526249]: Long tail cancellation due to optimal threshold Dec 30 15:54:41.139084886 (600h25m41.010167921s) [noise=true , address=77.90.31.19:4854 , piece_id=4W27FHSBO3L2TSAZL2JFCTUPHYV7RI5XDQ4SJWOCRCD5V4K4IK2Q, seg_pos={0 0}, upload=526249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:16HwqYhJnhseWEhzzv4QreZ953rB5cKxqFX8swEqaqiRXCRsAE): protocol: write tcp 188.166.225.170:48144->77.90.31.19:4854: use of closed network connection; write tcp 188.166.225.170:48144->77.90.31.19:4854: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48144->77.90.31.19:4854: use of closed network connection Dec 30 15:54:41.139093533 (600h25m41.010176568s) [noise=true , address=77.90.31.19:4854 , piece_id=4W27FHSBO3L2TSAZL2JFCTUPHYV7RI5XDQ4SJWOCRCD5V4K4IK2Q, seg_pos={0 0}, upload=526249]: Long tail cancellation due to optimal threshold Dec 30 15:54:41.139129352 (600h25m41.010212397s) [noise=true , address=145.79.227.43:4189 , piece_id=DYHLPLLBNG6B3FXERO33ZDDSHFUHQIIYGQZJFE2UMJXMLUBYC3KQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7): protocol: expected piece hash; context canceled; EOF Dec 30 15:54:41.139138429 (600h25m41.010221484s) [noise=true , address=145.79.227.43:4189 , piece_id=DYHLPLLBNG6B3FXERO33ZDDSHFUHQIIYGQZJFE2UMJXMLUBYC3KQ, seg_pos={0 0}, upload=526249]: Long tail cancellation due to optimal threshold Dec 30 15:54:41.139238890 (600h25m41.010321925s) [noise=true , address=64.49.13.169:4713 , piece_id=KI4XABHC36ADSOANW3HVM3TCWZ4ZXXXNP7MWJVESDKD5WO26DJHQ, seg_pos={0 0}, upload=526249]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:17z3AiUfbxBX2eMDnUnqo1bbgojuQxJro7PsEaPjYzQjHQ3D6K): protocol: write tcp 188.166.225.170:35558->64.49.13.169:4713: use of closed network connection; write tcp 188.166.225.170:35558->64.49.13.169:4713: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35558->64.49.13.169:4713: use of closed network connection Dec 30 15:54:41.139244420 (600h25m41.010327455s) [noise=true , address=64.49.13.169:4713 , piece_id=KI4XABHC36ADSOANW3HVM3TCWZ4ZXXXNP7MWJVESDKD5WO26DJHQ, seg_pos={0 0}, upload=526249]: Long tail cancellation due to optimal threshold Dec 30 15:54:41.139323681 (600h25m41.010406716s) [seg_pos={0 0}, upload=526249]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 15:54:41.139392121 (600h25m41.010475166s) [upload=526249]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 15:54:41.139396971 (600h25m41.010480006s) [upload=526249]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 15:54:41.438011559 (600h25m41.309094604s) [upload=526249]: Done uploading segments... Dec 30 15:54:41.438036206 (600h25m41.309119251s) [upload=526249]: Upload finished. err: <nil> Duration: 1m0.003169295s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 43 out of 60 is failed (threshold=6) [X..XXXX.XX.X.X.X.XXXXXX.XXXX...XX.X.XXXXXXXXX.XXXXXXX...XXXX] duration: 5.001083721s trace: https://jaeger.dev.storj.io/trace/237fc96070c8cc7b FAIL (2025-12-30 15:55:31 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001130199s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [......X.X..........XX.......XX.XX..X....X.......X......X.X..] duration: 5.868204115s trace: https://jaeger.dev.storj.io/trace/54fff38414d25b0f FAIL (2025-12-30 15:55:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: AQGHSCVDMOODBX225O5SSOLNLJQYPAY2X7EJ7G5W6RP42ZJ4DGCQ) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: ZTXP5ESUQMWFEWTQUX2SMER7W2NJG5IISRWFAM3IKAKQFKJOGWSQ) context deadline exceeded ======LOGS====== Dec 30 15:53:33.440030169 (600h24m33.311113204s) [upload=526231]: Starting upload Dec 30 15:53:33.440113437 (600h24m33.311196492s) [upload=526231]: Done starting upload Dec 30 15:53:33.440168081 (600h24m33.311251116s) [upload=526231]: Uploading segments... Dec 30 15:53:33.440357231 (600h24m33.311440276s) [upload=526231]: Got next segment. Inline: false Dec 30 15:53:33.440411114 (600h24m33.311494159s) [upload=526231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 15:53:33.440416885 (600h24m33.31149992s) [upload=526231]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 15:53:33.684984430 (600h24m33.556067505s) [seg_pos={0 0}, upload=526231]: Begin upload segment... Dec 30 15:53:33.685033994 (600h24m33.556117029s) [seg_pos={0 0}, upload=526231]: Done begin upload segment. Dec 30 15:53:33.685081235 (600h24m33.55616427s) [noise=true , address=145.79.227.36:4519 , piece_id=5JAZZV45ZTNDSTJZIX2EI7RK7MCBXBQUV2PLSF2HBCRFCIJEA6OQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.685200161 (600h24m33.556283196s) [noise=true , address=145.79.227.23:4545 , piece_id=NHTKPJFEFGAJOEXUNJJ7TU6N27JJP3O5TFSMWF5I2SMGEYXWEAKQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.685709841 (600h24m33.556792887s) [noise=true , address=77.90.31.14:4664 , piece_id=YLXNEMZBXUYTFQ4P6ZFHPA6BIUH4TZ2OBXEQAJWYSCUX333L777A, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686084574 (600h24m33.557167619s) [noise=true , address=45.91.116.12:4227 , piece_id=5GQIOQC4VDIF3Z6J4TUPZG3XUC3AACZ53CPYB4JIBQD7F5BK2IFA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686094794 (600h24m33.55717783s) [noise=true , address=77.90.31.10:4517 , piece_id=5Y32J2NPA664L4ZGRDDYYX66FPWND4DNZ5OZXAADJ3LOXK3IA5BA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686429431 (600h24m33.557512476s) [noise=true , address=45.91.116.13:4153 , piece_id=AQGHSCVDMOODBX225O5SSOLNLJQYPAY2X7EJ7G5W6RP42ZJ4DGCQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686489927 (600h24m33.557572962s) [noise=true , address=45.91.116.8:4047 , piece_id=TLKGZQ2VDRAGBZCULZ35ZR6GEJXQEE2HZ35Q52XBEO5C4FTRFICA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686910547 (600h24m33.557993582s) [noise=true , address=145.79.227.37:4567 , piece_id=WSHVGPU7L73OET2M6WT7VPHGVRQV6ZCJRPAW7AEXBUQDFWQBNAUA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686946536 (600h24m33.558029581s) [noise=true , address=145.79.227.14:4838 , piece_id=YB4VPIAQTBJ6G5XFXDVJ3OL6YFAWMEBHHS63SEDX5U4HYKXT3KJQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.687266755 (600h24m33.55834979s) [noise=true , address=145.79.227.6:4257 , piece_id=53MG7UDQR3MJYNMQXY3IOPPKOQEOI3KFUTGHQUEHOKCNG5NTAWXA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.687625419 (600h24m33.558708464s) [noise=true , address=45.91.116.11:4120 , piece_id=ZTXP5ESUQMWFEWTQUX2SMER7W2NJG5IISRWFAM3IKAKQFKJOGWSQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.688010913 (600h24m33.559093948s) [noise=true , address=145.79.227.19:4077 , piece_id=JEX52J3QTGUWMY3FO6BD5IJV5UPUUCKVV4MYZAOAU3BWMKFSN3KQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.688360248 (600h24m33.559443283s) [noise=true , address=145.79.227.16:4910 , piece_id=UHDFUXEQCMN73AUB7WEJBG5HY2G4E25MQGSBNK6AKHMQMDZCJYRA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.688673685 (600h24m33.55975672s) [noise=true , address=145.79.227.48:4303 , piece_id=IO7FUTELAP2L3VVZEQN6N6QG42WMAOS7H5JEPFAST5YEKMKHYLXA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.689057035 (600h24m33.56014007s) [noise=true , address=145.79.227.2:4114 , piece_id=3MJV2XXBY776ZUIO2KU4MEP37SP6P2NYFO255DCI6IFUZWUAGOWQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.690890025 (600h24m33.56197308s) [noise=true , address=145.79.227.3:4150 , piece_id=BUBUCPH3LTJO2CR3IVIXEOJOVS422KQT6L2Z22OBZP6DFPCK5KAA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:35.401137462 (600h24m35.272220507s) [upload=526231]: Next returned nil segment Dec 30 15:53:35.401154925 (600h24m35.27223797s) [upload=526231]: Waiting for error group managing segments... Dec 30 15:53:35.673442202 (600h24m35.544525237s) [noise=true , address=145.79.227.23:4545 , piece_id=NHTKPJFEFGAJOEXUNJJ7TU6N27JJP3O5TFSMWF5I2SMGEYXWEAKQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.674880822 (600h24m35.545963857s) [noise=true , address=145.79.227.2:4114 , piece_id=3MJV2XXBY776ZUIO2KU4MEP37SP6P2NYFO255DCI6IFUZWUAGOWQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.688801769 (600h24m35.559884814s) [noise=true , address=145.79.227.36:4519 , piece_id=5JAZZV45ZTNDSTJZIX2EI7RK7MCBXBQUV2PLSF2HBCRFCIJEA6OQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.689704678 (600h24m35.560787713s) [noise=true , address=145.79.227.6:4257 , piece_id=53MG7UDQR3MJYNMQXY3IOPPKOQEOI3KFUTGHQUEHOKCNG5NTAWXA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.690785346 (600h24m35.561868381s) [noise=true , address=145.79.227.14:4838 , piece_id=YB4VPIAQTBJ6G5XFXDVJ3OL6YFAWMEBHHS63SEDX5U4HYKXT3KJQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.692882529 (600h24m35.563965564s) [noise=true , address=145.79.227.37:4567 , piece_id=WSHVGPU7L73OET2M6WT7VPHGVRQV6ZCJRPAW7AEXBUQDFWQBNAUA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.696575461 (600h24m35.567658506s) [noise=true , address=145.79.227.16:4910 , piece_id=UHDFUXEQCMN73AUB7WEJBG5HY2G4E25MQGSBNK6AKHMQMDZCJYRA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.697864125 (600h24m35.56894716s) [noise=true , address=145.79.227.19:4077 , piece_id=JEX52J3QTGUWMY3FO6BD5IJV5UPUUCKVV4MYZAOAU3BWMKFSN3KQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.758530774 (600h24m35.629613819s) [noise=true , address=145.79.227.48:4303 , piece_id=IO7FUTELAP2L3VVZEQN6N6QG42WMAOS7H5JEPFAST5YEKMKHYLXA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.796785737 (600h24m35.667868772s) [noise=true , address=145.79.227.3:4150 , piece_id=BUBUCPH3LTJO2CR3IVIXEOJOVS422KQT6L2Z22OBZP6DFPCK5KAA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.029561278 (600h24m35.900644313s) [noise=true , address=77.90.31.10:4517 , piece_id=5Y32J2NPA664L4ZGRDDYYX66FPWND4DNZ5OZXAADJ3LOXK3IA5BA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.070210490 (600h24m35.941293536s) [noise=true , address=45.91.116.13:4153 , piece_id=AQGHSCVDMOODBX225O5SSOLNLJQYPAY2X7EJ7G5W6RP42ZJ4DGCQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.082591286 (600h24m35.953674331s) [noise=true , address=45.91.116.11:4120 , piece_id=ZTXP5ESUQMWFEWTQUX2SMER7W2NJG5IISRWFAM3IKAKQFKJOGWSQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.082610784 (600h24m35.953693819s) [seg_pos={0 0}, upload=526231]: Segment reached optimal threshold of 13 pieces. Dec 30 15:53:36.082858675 (600h24m35.953941711s) [noise=true , address=77.90.31.14:4664 , piece_id=YLXNEMZBXUYTFQ4P6ZFHPA6BIUH4TZ2OBXEQAJWYSCUX333L777A, seg_pos={0 0}, upload=526231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 30 15:53:36.082868053 (600h24m35.953951088s) [noise=true , address=77.90.31.14:4664 , piece_id=YLXNEMZBXUYTFQ4P6ZFHPA6BIUH4TZ2OBXEQAJWYSCUX333L777A, seg_pos={0 0}, upload=526231]: Long tail cancellation due to optimal threshold Dec 30 15:53:36.082953826 (600h24m35.954036871s) [noise=true , address=45.91.116.12:4227 , piece_id=5GQIOQC4VDIF3Z6J4TUPZG3XUC3AACZ53CPYB4JIBQD7F5BK2IFA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 30 15:53:36.082969977 (600h24m35.954053012s) [noise=true , address=45.91.116.12:4227 , piece_id=5GQIOQC4VDIF3Z6J4TUPZG3XUC3AACZ53CPYB4JIBQD7F5BK2IFA, seg_pos={0 0}, upload=526231]: Long tail cancellation due to optimal threshold Dec 30 15:53:36.083266191 (600h24m35.954349226s) [noise=true , address=45.91.116.8:4047 , piece_id=TLKGZQ2VDRAGBZCULZ35ZR6GEJXQEE2HZ35Q52XBEO5C4FTRFICA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: read tcp 188.166.225.170:42816->45.91.116.8:4047: use of closed network connection; context canceled Dec 30 15:53:36.083274006 (600h24m35.954357041s) [noise=true , address=45.91.116.8:4047 , piece_id=TLKGZQ2VDRAGBZCULZ35ZR6GEJXQEE2HZ35Q52XBEO5C4FTRFICA, seg_pos={0 0}, upload=526231]: Long tail cancellation due to optimal threshold Dec 30 15:53:36.083296048 (600h24m35.954379083s) [seg_pos={0 0}, upload=526231]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 15:53:36.083385679 (600h24m35.954468714s) [upload=526231]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 15:53:36.083392341 (600h24m35.954475376s) [upload=526231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 15:53:36.379743600 (600h24m36.250826635s) [upload=526231]: Done uploading segments... Dec 30 15:53:36.379758638 (600h24m36.250841673s) [upload=526231]: Upload finished. err: <nil> Duration: 5.868230483s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 13.73727946s trace: https://jaeger.dev.storj.io/trace/547648a2cfce50e7 FAIL (2025-12-30 15:54:41 UTC) orbiter warning: context deadline exceeded Duration: 13.737328644s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....X............................X..XX..X..................] duration: 28.13734322s trace: https://jaeger.dev.storj.io/trace/3b27a789ef618091 FAIL (2025-12-30 15:54:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ise=true , address=145.79.227.5:4252 , piece_id=JRKV35AO742VMYZPN3WKPNRZKO2ZFUU53RS4MXF22HO5UC6BKCOQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.081840064 (600h10m41.952923099s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.082191573 (600h10m41.953274608s) [noise=true , address=145.79.227.16:4942 , piece_id=G5I53KG36VHVW5LICAHOHRXTQVQ67ZFGJQNXRJMYNOIT7WB7XTEQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.082429897 (600h10m41.953512932s) [noise=true , address=145.79.227.29:4968 , piece_id=PMTNKRHI6YWMJPZ2R7RPIF7Q6CFLFKBH5OTHWCX4RNGIAFURVMOQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.082680584 (600h10m41.953763619s) [noise=true , address=145.79.227.33:4892 , piece_id=RSEJC3XHFACGLV6YBHYM7QBCU7MMYST53FJO4ITLA2C3IHBZCRTQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.082917976 (600h10m41.954001011s) [noise=true , address=77.90.31.12:4596 , piece_id=TQVNHTTAWVUCUIHSHWWH27MFJNOE2MWLJCRMTPUCREML7DT3RUYQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.083149276 (600h10m41.954232311s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.083393431 (600h10m41.954476466s) [noise=true , address=145.79.227.35:4508 , piece_id=KJJFNTPYUUHLFFCGHD3KDTN3DK6C2R5YRLWJDMF6C4UGSMFYVFBA, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.083842486 (600h10m41.954925521s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.086214362 (600h10m41.957297397s) [noise=true , address=145.79.227.49:4367 , piece_id=QUI5FSOTJVIHT67I3F2YDB5IPKSYEYJTDRKA44UM355S5FSNNLUA, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.086612852 (600h10m41.957695887s) [noise=true , address=77.90.31.13:4652 , piece_id=5VR2F6XUEYGX5KULBIL6V3YGH7Q3T5UYFJTS5RGWN3B7F7TVMHEQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.087061346 (600h10m41.958144381s) [noise=true , address=77.90.31.11:4588 , piece_id=WY5XS4EPFHDLCT3XEQQKBO2M5GLVTBNDZOKPTFPVZV6EALPMNSHA, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.087452310 (600h10m41.958535345s) [noise=true , address=145.79.227.30:4768 , piece_id=2KZKXIYGSDES5Q5W2PVAACJYU6ANKQNDP2VQEBXM7QKXOT6QLSBA, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.087903780 (600h10m41.958986825s) [noise=true , address=64.49.13.27:4957 , piece_id=ETFKCNNSFAVJWNB3J3U4LPRDGBILNNA5CK7FR7X5UHONPZHXFBLQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.088521366 (600h10m41.959604411s) [noise=true , address=193.149.164.143:4698 , piece_id=F6B3UPOKCSD7ZXQ4SYKOAALEBA4FSTQ2ELGMGCU7TMWVQZDZB2SQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.088618600 (600h10m41.959701645s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:43.511091675 (600h10m43.38217472s) [upload=526030]: Next returned nil segment Dec 30 15:39:43.511109379 (600h10m43.382192424s) [upload=526030]: Waiting for error group managing segments... Dec 30 15:39:44.017333248 (600h10m43.888416283s) [noise=true , address=145.79.227.5:4252 , piece_id=JRKV35AO742VMYZPN3WKPNRZKO2ZFUU53RS4MXF22HO5UC6BKCOQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.069300150 (600h10m43.940383186s) [noise=true , address=64.49.13.27:4957 , piece_id=ETFKCNNSFAVJWNB3J3U4LPRDGBILNNA5CK7FR7X5UHONPZHXFBLQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.126868228 (600h10m43.997951273s) [noise=true , address=77.90.31.17:4800 , piece_id=PYZVLAZZKWP6UOL6HJVVWSEJOXCJWK5IAMWJPRODOKCLRZE37BAQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.146896235 (600h10m44.01797927s) [noise=true , address=145.79.227.29:4968 , piece_id=PMTNKRHI6YWMJPZ2R7RPIF7Q6CFLFKBH5OTHWCX4RNGIAFURVMOQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.166221683 (600h10m44.037304718s) [noise=true , address=77.90.31.12:4596 , piece_id=TQVNHTTAWVUCUIHSHWWH27MFJNOE2MWLJCRMTPUCREML7DT3RUYQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.234530763 (600h10m44.105613808s) [noise=true , address=77.90.31.13:4652 , piece_id=5VR2F6XUEYGX5KULBIL6V3YGH7Q3T5UYFJTS5RGWN3B7F7TVMHEQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.489468910 (600h10m44.360551955s) [noise=true , address=145.79.227.16:4942 , piece_id=G5I53KG36VHVW5LICAHOHRXTQVQ67ZFGJQNXRJMYNOIT7WB7XTEQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.802041331 (600h10m44.673124366s) [noise=true , address=193.149.164.143:4698 , piece_id=F6B3UPOKCSD7ZXQ4SYKOAALEBA4FSTQ2ELGMGCU7TMWVQZDZB2SQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.822093824 (600h10m44.693176859s) [noise=true , address=145.79.227.49:4367 , piece_id=QUI5FSOTJVIHT67I3F2YDB5IPKSYEYJTDRKA44UM355S5FSNNLUA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.867276618 (600h10m44.738359653s) [noise=true , address=145.79.227.30:4768 , piece_id=2KZKXIYGSDES5Q5W2PVAACJYU6ANKQNDP2VQEBXM7QKXOT6QLSBA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.881025168 (600h10m44.752108203s) [noise=true , address=145.79.227.44:4199 , piece_id=AJKADWXHHMCPE6V52UMJJPCHOBBL7QDU7DF337LB6Q3YDHEYUUQQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.899437537 (600h10m44.770520572s) [noise=true , address=77.90.31.11:4588 , piece_id=WY5XS4EPFHDLCT3XEQQKBO2M5GLVTBNDZOKPTFPVZV6EALPMNSHA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.941939747 (600h10m44.813022782s) [noise=true , address=145.79.227.33:4892 , piece_id=RSEJC3XHFACGLV6YBHYM7QBCU7MMYST53FJO4ITLA2C3IHBZCRTQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.980395983 (600h10m44.851479029s) [noise=true , address=145.79.227.35:4508 , piece_id=KJJFNTPYUUHLFFCGHD3KDTN3DK6C2R5YRLWJDMF6C4UGSMFYVFBA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.980415871 (600h10m44.851498916s) [seg_pos={0 0}, upload=526030]: Segment reached optimal threshold of 14 pieces. Dec 30 15:39:44.980578861 (600h10m44.851661896s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection Dec 30 15:39:44.980584171 (600h10m44.851667206s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold Dec 30 15:39:44.980730901 (600h10m44.851813936s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 30 15:39:44.980747002 (600h10m44.851830037s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold Dec 30 15:39:44.980793770 (600h10m44.851876806s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127NKToES711diH4f8TFJbwERq3NhhFEFrQyi7Mka3VVLsamuzp): protocol: expected piece hash; context canceled; EOF Dec 30 15:39:44.980799651 (600h10m44.851882687s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold Dec 30 15:39:44.980930771 (600h10m44.852013806s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nnXd911ApLeiKnc4LgDEUNnnkxtnLEdwzP48BhuhyEykacV6W): protocol: expected piece hash; context canceled; EOF Dec 30 15:39:44.980937775 (600h10m44.85202081s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold Dec 30 15:39:44.980956481 (600h10m44.852039526s) [seg_pos={0 0}, upload=526030]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 15:39:44.981039398 (600h10m44.852122433s) [upload=526030]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 15:39:44.981056070 (600h10m44.852139105s) [upload=526030]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 15:39:45.290276710 (600h10m45.161359765s) [upload=526030]: Done uploading segments... Dec 30 15:39:45.290299182 (600h10m45.161382227s) [upload=526030]: Upload finished. err: <nil> Duration: 28.13735873s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 43 out of 60 is failed (threshold=6) [XX..XXXX.XX.X.X.X.XXXXXX.XXXX...XX.X.XXXXXXXXX.XXXXXXX...XXX] duration: 5.001342955s trace: https://jaeger.dev.storj.io/trace/5f18ac4558a1358d FAIL (2025-12-30 15:54:28 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001360888s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.......X.X..........XX.......XX.XX..X....X.......X......X.X.] duration: 5.800667553s trace: https://jaeger.dev.storj.io/trace/54fff38414d25b0f FAIL (2025-12-30 15:54:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: AQGHSCVDMOODBX225O5SSOLNLJQYPAY2X7EJ7G5W6RP42ZJ4DGCQ) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: ZTXP5ESUQMWFEWTQUX2SMER7W2NJG5IISRWFAM3IKAKQFKJOGWSQ) context deadline exceeded ======LOGS====== Dec 30 15:53:33.440030169 (600h24m33.311113204s) [upload=526231]: Starting upload Dec 30 15:53:33.440113437 (600h24m33.311196492s) [upload=526231]: Done starting upload Dec 30 15:53:33.440168081 (600h24m33.311251116s) [upload=526231]: Uploading segments... Dec 30 15:53:33.440357231 (600h24m33.311440276s) [upload=526231]: Got next segment. Inline: false Dec 30 15:53:33.440411114 (600h24m33.311494159s) [upload=526231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 15:53:33.440416885 (600h24m33.31149992s) [upload=526231]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 15:53:33.684984430 (600h24m33.556067505s) [seg_pos={0 0}, upload=526231]: Begin upload segment... Dec 30 15:53:33.685033994 (600h24m33.556117029s) [seg_pos={0 0}, upload=526231]: Done begin upload segment. Dec 30 15:53:33.685081235 (600h24m33.55616427s) [noise=true , address=145.79.227.36:4519 , piece_id=5JAZZV45ZTNDSTJZIX2EI7RK7MCBXBQUV2PLSF2HBCRFCIJEA6OQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.685200161 (600h24m33.556283196s) [noise=true , address=145.79.227.23:4545 , piece_id=NHTKPJFEFGAJOEXUNJJ7TU6N27JJP3O5TFSMWF5I2SMGEYXWEAKQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.685709841 (600h24m33.556792887s) [noise=true , address=77.90.31.14:4664 , piece_id=YLXNEMZBXUYTFQ4P6ZFHPA6BIUH4TZ2OBXEQAJWYSCUX333L777A, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686084574 (600h24m33.557167619s) [noise=true , address=45.91.116.12:4227 , piece_id=5GQIOQC4VDIF3Z6J4TUPZG3XUC3AACZ53CPYB4JIBQD7F5BK2IFA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686094794 (600h24m33.55717783s) [noise=true , address=77.90.31.10:4517 , piece_id=5Y32J2NPA664L4ZGRDDYYX66FPWND4DNZ5OZXAADJ3LOXK3IA5BA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686429431 (600h24m33.557512476s) [noise=true , address=45.91.116.13:4153 , piece_id=AQGHSCVDMOODBX225O5SSOLNLJQYPAY2X7EJ7G5W6RP42ZJ4DGCQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686489927 (600h24m33.557572962s) [noise=true , address=45.91.116.8:4047 , piece_id=TLKGZQ2VDRAGBZCULZ35ZR6GEJXQEE2HZ35Q52XBEO5C4FTRFICA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686910547 (600h24m33.557993582s) [noise=true , address=145.79.227.37:4567 , piece_id=WSHVGPU7L73OET2M6WT7VPHGVRQV6ZCJRPAW7AEXBUQDFWQBNAUA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686946536 (600h24m33.558029581s) [noise=true , address=145.79.227.14:4838 , piece_id=YB4VPIAQTBJ6G5XFXDVJ3OL6YFAWMEBHHS63SEDX5U4HYKXT3KJQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.687266755 (600h24m33.55834979s) [noise=true , address=145.79.227.6:4257 , piece_id=53MG7UDQR3MJYNMQXY3IOPPKOQEOI3KFUTGHQUEHOKCNG5NTAWXA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.687625419 (600h24m33.558708464s) [noise=true , address=45.91.116.11:4120 , piece_id=ZTXP5ESUQMWFEWTQUX2SMER7W2NJG5IISRWFAM3IKAKQFKJOGWSQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.688010913 (600h24m33.559093948s) [noise=true , address=145.79.227.19:4077 , piece_id=JEX52J3QTGUWMY3FO6BD5IJV5UPUUCKVV4MYZAOAU3BWMKFSN3KQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.688360248 (600h24m33.559443283s) [noise=true , address=145.79.227.16:4910 , piece_id=UHDFUXEQCMN73AUB7WEJBG5HY2G4E25MQGSBNK6AKHMQMDZCJYRA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.688673685 (600h24m33.55975672s) [noise=true , address=145.79.227.48:4303 , piece_id=IO7FUTELAP2L3VVZEQN6N6QG42WMAOS7H5JEPFAST5YEKMKHYLXA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.689057035 (600h24m33.56014007s) [noise=true , address=145.79.227.2:4114 , piece_id=3MJV2XXBY776ZUIO2KU4MEP37SP6P2NYFO255DCI6IFUZWUAGOWQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.690890025 (600h24m33.56197308s) [noise=true , address=145.79.227.3:4150 , piece_id=BUBUCPH3LTJO2CR3IVIXEOJOVS422KQT6L2Z22OBZP6DFPCK5KAA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:35.401137462 (600h24m35.272220507s) [upload=526231]: Next returned nil segment Dec 30 15:53:35.401154925 (600h24m35.27223797s) [upload=526231]: Waiting for error group managing segments... Dec 30 15:53:35.673442202 (600h24m35.544525237s) [noise=true , address=145.79.227.23:4545 , piece_id=NHTKPJFEFGAJOEXUNJJ7TU6N27JJP3O5TFSMWF5I2SMGEYXWEAKQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.674880822 (600h24m35.545963857s) [noise=true , address=145.79.227.2:4114 , piece_id=3MJV2XXBY776ZUIO2KU4MEP37SP6P2NYFO255DCI6IFUZWUAGOWQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.688801769 (600h24m35.559884814s) [noise=true , address=145.79.227.36:4519 , piece_id=5JAZZV45ZTNDSTJZIX2EI7RK7MCBXBQUV2PLSF2HBCRFCIJEA6OQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.689704678 (600h24m35.560787713s) [noise=true , address=145.79.227.6:4257 , piece_id=53MG7UDQR3MJYNMQXY3IOPPKOQEOI3KFUTGHQUEHOKCNG5NTAWXA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.690785346 (600h24m35.561868381s) [noise=true , address=145.79.227.14:4838 , piece_id=YB4VPIAQTBJ6G5XFXDVJ3OL6YFAWMEBHHS63SEDX5U4HYKXT3KJQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.692882529 (600h24m35.563965564s) [noise=true , address=145.79.227.37:4567 , piece_id=WSHVGPU7L73OET2M6WT7VPHGVRQV6ZCJRPAW7AEXBUQDFWQBNAUA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.696575461 (600h24m35.567658506s) [noise=true , address=145.79.227.16:4910 , piece_id=UHDFUXEQCMN73AUB7WEJBG5HY2G4E25MQGSBNK6AKHMQMDZCJYRA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.697864125 (600h24m35.56894716s) [noise=true , address=145.79.227.19:4077 , piece_id=JEX52J3QTGUWMY3FO6BD5IJV5UPUUCKVV4MYZAOAU3BWMKFSN3KQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.758530774 (600h24m35.629613819s) [noise=true , address=145.79.227.48:4303 , piece_id=IO7FUTELAP2L3VVZEQN6N6QG42WMAOS7H5JEPFAST5YEKMKHYLXA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.796785737 (600h24m35.667868772s) [noise=true , address=145.79.227.3:4150 , piece_id=BUBUCPH3LTJO2CR3IVIXEOJOVS422KQT6L2Z22OBZP6DFPCK5KAA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.029561278 (600h24m35.900644313s) [noise=true , address=77.90.31.10:4517 , piece_id=5Y32J2NPA664L4ZGRDDYYX66FPWND4DNZ5OZXAADJ3LOXK3IA5BA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.070210490 (600h24m35.941293536s) [noise=true , address=45.91.116.13:4153 , piece_id=AQGHSCVDMOODBX225O5SSOLNLJQYPAY2X7EJ7G5W6RP42ZJ4DGCQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.082591286 (600h24m35.953674331s) [noise=true , address=45.91.116.11:4120 , piece_id=ZTXP5ESUQMWFEWTQUX2SMER7W2NJG5IISRWFAM3IKAKQFKJOGWSQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.082610784 (600h24m35.953693819s) [seg_pos={0 0}, upload=526231]: Segment reached optimal threshold of 13 pieces. Dec 30 15:53:36.082858675 (600h24m35.953941711s) [noise=true , address=77.90.31.14:4664 , piece_id=YLXNEMZBXUYTFQ4P6ZFHPA6BIUH4TZ2OBXEQAJWYSCUX333L777A, seg_pos={0 0}, upload=526231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 30 15:53:36.082868053 (600h24m35.953951088s) [noise=true , address=77.90.31.14:4664 , piece_id=YLXNEMZBXUYTFQ4P6ZFHPA6BIUH4TZ2OBXEQAJWYSCUX333L777A, seg_pos={0 0}, upload=526231]: Long tail cancellation due to optimal threshold Dec 30 15:53:36.082953826 (600h24m35.954036871s) [noise=true , address=45.91.116.12:4227 , piece_id=5GQIOQC4VDIF3Z6J4TUPZG3XUC3AACZ53CPYB4JIBQD7F5BK2IFA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 30 15:53:36.082969977 (600h24m35.954053012s) [noise=true , address=45.91.116.12:4227 , piece_id=5GQIOQC4VDIF3Z6J4TUPZG3XUC3AACZ53CPYB4JIBQD7F5BK2IFA, seg_pos={0 0}, upload=526231]: Long tail cancellation due to optimal threshold Dec 30 15:53:36.083266191 (600h24m35.954349226s) [noise=true , address=45.91.116.8:4047 , piece_id=TLKGZQ2VDRAGBZCULZ35ZR6GEJXQEE2HZ35Q52XBEO5C4FTRFICA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: read tcp 188.166.225.170:42816->45.91.116.8:4047: use of closed network connection; context canceled Dec 30 15:53:36.083274006 (600h24m35.954357041s) [noise=true , address=45.91.116.8:4047 , piece_id=TLKGZQ2VDRAGBZCULZ35ZR6GEJXQEE2HZ35Q52XBEO5C4FTRFICA, seg_pos={0 0}, upload=526231]: Long tail cancellation due to optimal threshold Dec 30 15:53:36.083296048 (600h24m35.954379083s) [seg_pos={0 0}, upload=526231]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 15:53:36.083385679 (600h24m35.954468714s) [upload=526231]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 15:53:36.083392341 (600h24m35.954475376s) [upload=526231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 15:53:36.379743600 (600h24m36.250826635s) [upload=526231]: Done uploading segments... Dec 30 15:53:36.379758638 (600h24m36.250841673s) [upload=526231]: Upload finished. err: <nil> Duration: 5.800695606s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 12.987670274s trace: https://jaeger.dev.storj.io/trace/547648a2cfce50e7 FAIL (2025-12-30 15:54:24 UTC) orbiter warning: context deadline exceeded Duration: 12.987791135s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 43 out of 60 is failed (threshold=6) [XXX..XXXX.XX.X.X.X.XXXXXX.XXXX...XX.X.XXXXXXXXX.XXXXXXX...XX] duration: 5.001476757s trace: https://jaeger.dev.storj.io/trace/6c0c468667f04732 FAIL (2025-12-30 15:54:12 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001516243s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [......X............................X..XX..X.................] duration: 35.144930852s trace: https://jaeger.dev.storj.io/trace/3b27a789ef618091 FAIL (2025-12-30 15:54:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ise=true , address=145.79.227.5:4252 , piece_id=JRKV35AO742VMYZPN3WKPNRZKO2ZFUU53RS4MXF22HO5UC6BKCOQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.081840064 (600h10m41.952923099s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.082191573 (600h10m41.953274608s) [noise=true , address=145.79.227.16:4942 , piece_id=G5I53KG36VHVW5LICAHOHRXTQVQ67ZFGJQNXRJMYNOIT7WB7XTEQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.082429897 (600h10m41.953512932s) [noise=true , address=145.79.227.29:4968 , piece_id=PMTNKRHI6YWMJPZ2R7RPIF7Q6CFLFKBH5OTHWCX4RNGIAFURVMOQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.082680584 (600h10m41.953763619s) [noise=true , address=145.79.227.33:4892 , piece_id=RSEJC3XHFACGLV6YBHYM7QBCU7MMYST53FJO4ITLA2C3IHBZCRTQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.082917976 (600h10m41.954001011s) [noise=true , address=77.90.31.12:4596 , piece_id=TQVNHTTAWVUCUIHSHWWH27MFJNOE2MWLJCRMTPUCREML7DT3RUYQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.083149276 (600h10m41.954232311s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.083393431 (600h10m41.954476466s) [noise=true , address=145.79.227.35:4508 , piece_id=KJJFNTPYUUHLFFCGHD3KDTN3DK6C2R5YRLWJDMF6C4UGSMFYVFBA, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.083842486 (600h10m41.954925521s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.086214362 (600h10m41.957297397s) [noise=true , address=145.79.227.49:4367 , piece_id=QUI5FSOTJVIHT67I3F2YDB5IPKSYEYJTDRKA44UM355S5FSNNLUA, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.086612852 (600h10m41.957695887s) [noise=true , address=77.90.31.13:4652 , piece_id=5VR2F6XUEYGX5KULBIL6V3YGH7Q3T5UYFJTS5RGWN3B7F7TVMHEQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.087061346 (600h10m41.958144381s) [noise=true , address=77.90.31.11:4588 , piece_id=WY5XS4EPFHDLCT3XEQQKBO2M5GLVTBNDZOKPTFPVZV6EALPMNSHA, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.087452310 (600h10m41.958535345s) [noise=true , address=145.79.227.30:4768 , piece_id=2KZKXIYGSDES5Q5W2PVAACJYU6ANKQNDP2VQEBXM7QKXOT6QLSBA, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.087903780 (600h10m41.958986825s) [noise=true , address=64.49.13.27:4957 , piece_id=ETFKCNNSFAVJWNB3J3U4LPRDGBILNNA5CK7FR7X5UHONPZHXFBLQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.088521366 (600h10m41.959604411s) [noise=true , address=193.149.164.143:4698 , piece_id=F6B3UPOKCSD7ZXQ4SYKOAALEBA4FSTQ2ELGMGCU7TMWVQZDZB2SQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.088618600 (600h10m41.959701645s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:43.511091675 (600h10m43.38217472s) [upload=526030]: Next returned nil segment Dec 30 15:39:43.511109379 (600h10m43.382192424s) [upload=526030]: Waiting for error group managing segments... Dec 30 15:39:44.017333248 (600h10m43.888416283s) [noise=true , address=145.79.227.5:4252 , piece_id=JRKV35AO742VMYZPN3WKPNRZKO2ZFUU53RS4MXF22HO5UC6BKCOQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.069300150 (600h10m43.940383186s) [noise=true , address=64.49.13.27:4957 , piece_id=ETFKCNNSFAVJWNB3J3U4LPRDGBILNNA5CK7FR7X5UHONPZHXFBLQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.126868228 (600h10m43.997951273s) [noise=true , address=77.90.31.17:4800 , piece_id=PYZVLAZZKWP6UOL6HJVVWSEJOXCJWK5IAMWJPRODOKCLRZE37BAQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.146896235 (600h10m44.01797927s) [noise=true , address=145.79.227.29:4968 , piece_id=PMTNKRHI6YWMJPZ2R7RPIF7Q6CFLFKBH5OTHWCX4RNGIAFURVMOQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.166221683 (600h10m44.037304718s) [noise=true , address=77.90.31.12:4596 , piece_id=TQVNHTTAWVUCUIHSHWWH27MFJNOE2MWLJCRMTPUCREML7DT3RUYQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.234530763 (600h10m44.105613808s) [noise=true , address=77.90.31.13:4652 , piece_id=5VR2F6XUEYGX5KULBIL6V3YGH7Q3T5UYFJTS5RGWN3B7F7TVMHEQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.489468910 (600h10m44.360551955s) [noise=true , address=145.79.227.16:4942 , piece_id=G5I53KG36VHVW5LICAHOHRXTQVQ67ZFGJQNXRJMYNOIT7WB7XTEQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.802041331 (600h10m44.673124366s) [noise=true , address=193.149.164.143:4698 , piece_id=F6B3UPOKCSD7ZXQ4SYKOAALEBA4FSTQ2ELGMGCU7TMWVQZDZB2SQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.822093824 (600h10m44.693176859s) [noise=true , address=145.79.227.49:4367 , piece_id=QUI5FSOTJVIHT67I3F2YDB5IPKSYEYJTDRKA44UM355S5FSNNLUA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.867276618 (600h10m44.738359653s) [noise=true , address=145.79.227.30:4768 , piece_id=2KZKXIYGSDES5Q5W2PVAACJYU6ANKQNDP2VQEBXM7QKXOT6QLSBA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.881025168 (600h10m44.752108203s) [noise=true , address=145.79.227.44:4199 , piece_id=AJKADWXHHMCPE6V52UMJJPCHOBBL7QDU7DF337LB6Q3YDHEYUUQQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.899437537 (600h10m44.770520572s) [noise=true , address=77.90.31.11:4588 , piece_id=WY5XS4EPFHDLCT3XEQQKBO2M5GLVTBNDZOKPTFPVZV6EALPMNSHA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.941939747 (600h10m44.813022782s) [noise=true , address=145.79.227.33:4892 , piece_id=RSEJC3XHFACGLV6YBHYM7QBCU7MMYST53FJO4ITLA2C3IHBZCRTQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.980395983 (600h10m44.851479029s) [noise=true , address=145.79.227.35:4508 , piece_id=KJJFNTPYUUHLFFCGHD3KDTN3DK6C2R5YRLWJDMF6C4UGSMFYVFBA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.980415871 (600h10m44.851498916s) [seg_pos={0 0}, upload=526030]: Segment reached optimal threshold of 14 pieces. Dec 30 15:39:44.980578861 (600h10m44.851661896s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection Dec 30 15:39:44.980584171 (600h10m44.851667206s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold Dec 30 15:39:44.980730901 (600h10m44.851813936s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 30 15:39:44.980747002 (600h10m44.851830037s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold Dec 30 15:39:44.980793770 (600h10m44.851876806s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127NKToES711diH4f8TFJbwERq3NhhFEFrQyi7Mka3VVLsamuzp): protocol: expected piece hash; context canceled; EOF Dec 30 15:39:44.980799651 (600h10m44.851882687s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold Dec 30 15:39:44.980930771 (600h10m44.852013806s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nnXd911ApLeiKnc4LgDEUNnnkxtnLEdwzP48BhuhyEykacV6W): protocol: expected piece hash; context canceled; EOF Dec 30 15:39:44.980937775 (600h10m44.85202081s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold Dec 30 15:39:44.980956481 (600h10m44.852039526s) [seg_pos={0 0}, upload=526030]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 15:39:44.981039398 (600h10m44.852122433s) [upload=526030]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 15:39:44.981056070 (600h10m44.852139105s) [upload=526030]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 15:39:45.290276710 (600h10m45.161359765s) [upload=526030]: Done uploading segments... Dec 30 15:39:45.290299182 (600h10m45.161382227s) [upload=526030]: Upload finished. err: <nil> Duration: 35.144956292s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X.......X.X..........XX.......XX.XX..X....X.......X......X.X] duration: 30.003163921s trace: https://jaeger.dev.storj.io/trace/54fff38414d25b0f FAIL (2025-12-30 15:54:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: AQGHSCVDMOODBX225O5SSOLNLJQYPAY2X7EJ7G5W6RP42ZJ4DGCQ) context deadline exceeded; piecestore: (Node ID: 1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8, Piece ID: ZTXP5ESUQMWFEWTQUX2SMER7W2NJG5IISRWFAM3IKAKQFKJOGWSQ) context deadline exceeded ======LOGS====== Dec 30 15:53:33.440030169 (600h24m33.311113204s) [upload=526231]: Starting upload Dec 30 15:53:33.440113437 (600h24m33.311196492s) [upload=526231]: Done starting upload Dec 30 15:53:33.440168081 (600h24m33.311251116s) [upload=526231]: Uploading segments... Dec 30 15:53:33.440357231 (600h24m33.311440276s) [upload=526231]: Got next segment. Inline: false Dec 30 15:53:33.440411114 (600h24m33.311494159s) [upload=526231]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 15:53:33.440416885 (600h24m33.31149992s) [upload=526231]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 15:53:33.684984430 (600h24m33.556067505s) [seg_pos={0 0}, upload=526231]: Begin upload segment... Dec 30 15:53:33.685033994 (600h24m33.556117029s) [seg_pos={0 0}, upload=526231]: Done begin upload segment. Dec 30 15:53:33.685081235 (600h24m33.55616427s) [noise=true , address=145.79.227.36:4519 , piece_id=5JAZZV45ZTNDSTJZIX2EI7RK7MCBXBQUV2PLSF2HBCRFCIJEA6OQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.685200161 (600h24m33.556283196s) [noise=true , address=145.79.227.23:4545 , piece_id=NHTKPJFEFGAJOEXUNJJ7TU6N27JJP3O5TFSMWF5I2SMGEYXWEAKQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.685709841 (600h24m33.556792887s) [noise=true , address=77.90.31.14:4664 , piece_id=YLXNEMZBXUYTFQ4P6ZFHPA6BIUH4TZ2OBXEQAJWYSCUX333L777A, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686084574 (600h24m33.557167619s) [noise=true , address=45.91.116.12:4227 , piece_id=5GQIOQC4VDIF3Z6J4TUPZG3XUC3AACZ53CPYB4JIBQD7F5BK2IFA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686094794 (600h24m33.55717783s) [noise=true , address=77.90.31.10:4517 , piece_id=5Y32J2NPA664L4ZGRDDYYX66FPWND4DNZ5OZXAADJ3LOXK3IA5BA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686429431 (600h24m33.557512476s) [noise=true , address=45.91.116.13:4153 , piece_id=AQGHSCVDMOODBX225O5SSOLNLJQYPAY2X7EJ7G5W6RP42ZJ4DGCQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686489927 (600h24m33.557572962s) [noise=true , address=45.91.116.8:4047 , piece_id=TLKGZQ2VDRAGBZCULZ35ZR6GEJXQEE2HZ35Q52XBEO5C4FTRFICA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686910547 (600h24m33.557993582s) [noise=true , address=145.79.227.37:4567 , piece_id=WSHVGPU7L73OET2M6WT7VPHGVRQV6ZCJRPAW7AEXBUQDFWQBNAUA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.686946536 (600h24m33.558029581s) [noise=true , address=145.79.227.14:4838 , piece_id=YB4VPIAQTBJ6G5XFXDVJ3OL6YFAWMEBHHS63SEDX5U4HYKXT3KJQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.687266755 (600h24m33.55834979s) [noise=true , address=145.79.227.6:4257 , piece_id=53MG7UDQR3MJYNMQXY3IOPPKOQEOI3KFUTGHQUEHOKCNG5NTAWXA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.687625419 (600h24m33.558708464s) [noise=true , address=45.91.116.11:4120 , piece_id=ZTXP5ESUQMWFEWTQUX2SMER7W2NJG5IISRWFAM3IKAKQFKJOGWSQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.688010913 (600h24m33.559093948s) [noise=true , address=145.79.227.19:4077 , piece_id=JEX52J3QTGUWMY3FO6BD5IJV5UPUUCKVV4MYZAOAU3BWMKFSN3KQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.688360248 (600h24m33.559443283s) [noise=true , address=145.79.227.16:4910 , piece_id=UHDFUXEQCMN73AUB7WEJBG5HY2G4E25MQGSBNK6AKHMQMDZCJYRA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.688673685 (600h24m33.55975672s) [noise=true , address=145.79.227.48:4303 , piece_id=IO7FUTELAP2L3VVZEQN6N6QG42WMAOS7H5JEPFAST5YEKMKHYLXA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.689057035 (600h24m33.56014007s) [noise=true , address=145.79.227.2:4114 , piece_id=3MJV2XXBY776ZUIO2KU4MEP37SP6P2NYFO255DCI6IFUZWUAGOWQ, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:33.690890025 (600h24m33.56197308s) [noise=true , address=145.79.227.3:4150 , piece_id=BUBUCPH3LTJO2CR3IVIXEOJOVS422KQT6L2Z22OBZP6DFPCK5KAA, seg_pos={0 0}, upload=526231]: Uploading piece... Dec 30 15:53:35.401137462 (600h24m35.272220507s) [upload=526231]: Next returned nil segment Dec 30 15:53:35.401154925 (600h24m35.27223797s) [upload=526231]: Waiting for error group managing segments... Dec 30 15:53:35.673442202 (600h24m35.544525237s) [noise=true , address=145.79.227.23:4545 , piece_id=NHTKPJFEFGAJOEXUNJJ7TU6N27JJP3O5TFSMWF5I2SMGEYXWEAKQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.674880822 (600h24m35.545963857s) [noise=true , address=145.79.227.2:4114 , piece_id=3MJV2XXBY776ZUIO2KU4MEP37SP6P2NYFO255DCI6IFUZWUAGOWQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.688801769 (600h24m35.559884814s) [noise=true , address=145.79.227.36:4519 , piece_id=5JAZZV45ZTNDSTJZIX2EI7RK7MCBXBQUV2PLSF2HBCRFCIJEA6OQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.689704678 (600h24m35.560787713s) [noise=true , address=145.79.227.6:4257 , piece_id=53MG7UDQR3MJYNMQXY3IOPPKOQEOI3KFUTGHQUEHOKCNG5NTAWXA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.690785346 (600h24m35.561868381s) [noise=true , address=145.79.227.14:4838 , piece_id=YB4VPIAQTBJ6G5XFXDVJ3OL6YFAWMEBHHS63SEDX5U4HYKXT3KJQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.692882529 (600h24m35.563965564s) [noise=true , address=145.79.227.37:4567 , piece_id=WSHVGPU7L73OET2M6WT7VPHGVRQV6ZCJRPAW7AEXBUQDFWQBNAUA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.696575461 (600h24m35.567658506s) [noise=true , address=145.79.227.16:4910 , piece_id=UHDFUXEQCMN73AUB7WEJBG5HY2G4E25MQGSBNK6AKHMQMDZCJYRA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.697864125 (600h24m35.56894716s) [noise=true , address=145.79.227.19:4077 , piece_id=JEX52J3QTGUWMY3FO6BD5IJV5UPUUCKVV4MYZAOAU3BWMKFSN3KQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.758530774 (600h24m35.629613819s) [noise=true , address=145.79.227.48:4303 , piece_id=IO7FUTELAP2L3VVZEQN6N6QG42WMAOS7H5JEPFAST5YEKMKHYLXA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:35.796785737 (600h24m35.667868772s) [noise=true , address=145.79.227.3:4150 , piece_id=BUBUCPH3LTJO2CR3IVIXEOJOVS422KQT6L2Z22OBZP6DFPCK5KAA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.029561278 (600h24m35.900644313s) [noise=true , address=77.90.31.10:4517 , piece_id=5Y32J2NPA664L4ZGRDDYYX66FPWND4DNZ5OZXAADJ3LOXK3IA5BA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.070210490 (600h24m35.941293536s) [noise=true , address=45.91.116.13:4153 , piece_id=AQGHSCVDMOODBX225O5SSOLNLJQYPAY2X7EJ7G5W6RP42ZJ4DGCQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.082591286 (600h24m35.953674331s) [noise=true , address=45.91.116.11:4120 , piece_id=ZTXP5ESUQMWFEWTQUX2SMER7W2NJG5IISRWFAM3IKAKQFKJOGWSQ, seg_pos={0 0}, upload=526231]: Done uploading piece. err: <nil> Dec 30 15:53:36.082610784 (600h24m35.953693819s) [seg_pos={0 0}, upload=526231]: Segment reached optimal threshold of 13 pieces. Dec 30 15:53:36.082858675 (600h24m35.953941711s) [noise=true , address=77.90.31.14:4664 , piece_id=YLXNEMZBXUYTFQ4P6ZFHPA6BIUH4TZ2OBXEQAJWYSCUX333L777A, seg_pos={0 0}, upload=526231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9): protocol: expected piece hash; context canceled; EOF Dec 30 15:53:36.082868053 (600h24m35.953951088s) [noise=true , address=77.90.31.14:4664 , piece_id=YLXNEMZBXUYTFQ4P6ZFHPA6BIUH4TZ2OBXEQAJWYSCUX333L777A, seg_pos={0 0}, upload=526231]: Long tail cancellation due to optimal threshold Dec 30 15:53:36.082953826 (600h24m35.954036871s) [noise=true , address=45.91.116.12:4227 , piece_id=5GQIOQC4VDIF3Z6J4TUPZG3XUC3AACZ53CPYB4JIBQD7F5BK2IFA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 30 15:53:36.082969977 (600h24m35.954053012s) [noise=true , address=45.91.116.12:4227 , piece_id=5GQIOQC4VDIF3Z6J4TUPZG3XUC3AACZ53CPYB4JIBQD7F5BK2IFA, seg_pos={0 0}, upload=526231]: Long tail cancellation due to optimal threshold Dec 30 15:53:36.083266191 (600h24m35.954349226s) [noise=true , address=45.91.116.8:4047 , piece_id=TLKGZQ2VDRAGBZCULZ35ZR6GEJXQEE2HZ35Q52XBEO5C4FTRFICA, seg_pos={0 0}, upload=526231]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: read tcp 188.166.225.170:42816->45.91.116.8:4047: use of closed network connection; context canceled Dec 30 15:53:36.083274006 (600h24m35.954357041s) [noise=true , address=45.91.116.8:4047 , piece_id=TLKGZQ2VDRAGBZCULZ35ZR6GEJXQEE2HZ35Q52XBEO5C4FTRFICA, seg_pos={0 0}, upload=526231]: Long tail cancellation due to optimal threshold Dec 30 15:53:36.083296048 (600h24m35.954379083s) [seg_pos={0 0}, upload=526231]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 15:53:36.083385679 (600h24m35.954468714s) [upload=526231]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 15:53:36.083392341 (600h24m35.954475376s) [upload=526231]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 15:53:36.379743600 (600h24m36.250826635s) [upload=526231]: Done uploading segments... Dec 30 15:53:36.379758638 (600h24m36.250841673s) [upload=526231]: Upload finished. err: <nil> Duration: 30.003196493s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 13.979532312s trace: https://jaeger.dev.storj.io/trace/547648a2cfce50e7 FAIL (2025-12-30 15:53:37 UTC) orbiter warning: context deadline exceeded Duration: 13.979556889s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 43 out of 60 is failed (threshold=6) [XXXX..XXXX.XX.X.X.X.XXXXXX.XXXX...XX.X.XXXXXXXXX.XXXXXXX...X] duration: 5.001690535s trace: https://jaeger.dev.storj.io/trace/6a6aa0ea4122760d FAIL (2025-12-30 15:53:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001714741s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......X............................X..XX..X................] duration: 45.231145574s trace: https://jaeger.dev.storj.io/trace/3b27a789ef618091 FAIL (2025-12-30 15:53:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ise=true , address=145.79.227.5:4252 , piece_id=JRKV35AO742VMYZPN3WKPNRZKO2ZFUU53RS4MXF22HO5UC6BKCOQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.081840064 (600h10m41.952923099s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.082191573 (600h10m41.953274608s) [noise=true , address=145.79.227.16:4942 , piece_id=G5I53KG36VHVW5LICAHOHRXTQVQ67ZFGJQNXRJMYNOIT7WB7XTEQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.082429897 (600h10m41.953512932s) [noise=true , address=145.79.227.29:4968 , piece_id=PMTNKRHI6YWMJPZ2R7RPIF7Q6CFLFKBH5OTHWCX4RNGIAFURVMOQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.082680584 (600h10m41.953763619s) [noise=true , address=145.79.227.33:4892 , piece_id=RSEJC3XHFACGLV6YBHYM7QBCU7MMYST53FJO4ITLA2C3IHBZCRTQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.082917976 (600h10m41.954001011s) [noise=true , address=77.90.31.12:4596 , piece_id=TQVNHTTAWVUCUIHSHWWH27MFJNOE2MWLJCRMTPUCREML7DT3RUYQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.083149276 (600h10m41.954232311s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.083393431 (600h10m41.954476466s) [noise=true , address=145.79.227.35:4508 , piece_id=KJJFNTPYUUHLFFCGHD3KDTN3DK6C2R5YRLWJDMF6C4UGSMFYVFBA, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.083842486 (600h10m41.954925521s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.086214362 (600h10m41.957297397s) [noise=true , address=145.79.227.49:4367 , piece_id=QUI5FSOTJVIHT67I3F2YDB5IPKSYEYJTDRKA44UM355S5FSNNLUA, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.086612852 (600h10m41.957695887s) [noise=true , address=77.90.31.13:4652 , piece_id=5VR2F6XUEYGX5KULBIL6V3YGH7Q3T5UYFJTS5RGWN3B7F7TVMHEQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.087061346 (600h10m41.958144381s) [noise=true , address=77.90.31.11:4588 , piece_id=WY5XS4EPFHDLCT3XEQQKBO2M5GLVTBNDZOKPTFPVZV6EALPMNSHA, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.087452310 (600h10m41.958535345s) [noise=true , address=145.79.227.30:4768 , piece_id=2KZKXIYGSDES5Q5W2PVAACJYU6ANKQNDP2VQEBXM7QKXOT6QLSBA, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.087903780 (600h10m41.958986825s) [noise=true , address=64.49.13.27:4957 , piece_id=ETFKCNNSFAVJWNB3J3U4LPRDGBILNNA5CK7FR7X5UHONPZHXFBLQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.088521366 (600h10m41.959604411s) [noise=true , address=193.149.164.143:4698 , piece_id=F6B3UPOKCSD7ZXQ4SYKOAALEBA4FSTQ2ELGMGCU7TMWVQZDZB2SQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.088618600 (600h10m41.959701645s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:43.511091675 (600h10m43.38217472s) [upload=526030]: Next returned nil segment Dec 30 15:39:43.511109379 (600h10m43.382192424s) [upload=526030]: Waiting for error group managing segments... Dec 30 15:39:44.017333248 (600h10m43.888416283s) [noise=true , address=145.79.227.5:4252 , piece_id=JRKV35AO742VMYZPN3WKPNRZKO2ZFUU53RS4MXF22HO5UC6BKCOQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.069300150 (600h10m43.940383186s) [noise=true , address=64.49.13.27:4957 , piece_id=ETFKCNNSFAVJWNB3J3U4LPRDGBILNNA5CK7FR7X5UHONPZHXFBLQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.126868228 (600h10m43.997951273s) [noise=true , address=77.90.31.17:4800 , piece_id=PYZVLAZZKWP6UOL6HJVVWSEJOXCJWK5IAMWJPRODOKCLRZE37BAQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.146896235 (600h10m44.01797927s) [noise=true , address=145.79.227.29:4968 , piece_id=PMTNKRHI6YWMJPZ2R7RPIF7Q6CFLFKBH5OTHWCX4RNGIAFURVMOQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.166221683 (600h10m44.037304718s) [noise=true , address=77.90.31.12:4596 , piece_id=TQVNHTTAWVUCUIHSHWWH27MFJNOE2MWLJCRMTPUCREML7DT3RUYQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.234530763 (600h10m44.105613808s) [noise=true , address=77.90.31.13:4652 , piece_id=5VR2F6XUEYGX5KULBIL6V3YGH7Q3T5UYFJTS5RGWN3B7F7TVMHEQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.489468910 (600h10m44.360551955s) [noise=true , address=145.79.227.16:4942 , piece_id=G5I53KG36VHVW5LICAHOHRXTQVQ67ZFGJQNXRJMYNOIT7WB7XTEQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.802041331 (600h10m44.673124366s) [noise=true , address=193.149.164.143:4698 , piece_id=F6B3UPOKCSD7ZXQ4SYKOAALEBA4FSTQ2ELGMGCU7TMWVQZDZB2SQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.822093824 (600h10m44.693176859s) [noise=true , address=145.79.227.49:4367 , piece_id=QUI5FSOTJVIHT67I3F2YDB5IPKSYEYJTDRKA44UM355S5FSNNLUA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.867276618 (600h10m44.738359653s) [noise=true , address=145.79.227.30:4768 , piece_id=2KZKXIYGSDES5Q5W2PVAACJYU6ANKQNDP2VQEBXM7QKXOT6QLSBA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.881025168 (600h10m44.752108203s) [noise=true , address=145.79.227.44:4199 , piece_id=AJKADWXHHMCPE6V52UMJJPCHOBBL7QDU7DF337LB6Q3YDHEYUUQQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.899437537 (600h10m44.770520572s) [noise=true , address=77.90.31.11:4588 , piece_id=WY5XS4EPFHDLCT3XEQQKBO2M5GLVTBNDZOKPTFPVZV6EALPMNSHA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.941939747 (600h10m44.813022782s) [noise=true , address=145.79.227.33:4892 , piece_id=RSEJC3XHFACGLV6YBHYM7QBCU7MMYST53FJO4ITLA2C3IHBZCRTQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.980395983 (600h10m44.851479029s) [noise=true , address=145.79.227.35:4508 , piece_id=KJJFNTPYUUHLFFCGHD3KDTN3DK6C2R5YRLWJDMF6C4UGSMFYVFBA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.980415871 (600h10m44.851498916s) [seg_pos={0 0}, upload=526030]: Segment reached optimal threshold of 14 pieces. Dec 30 15:39:44.980578861 (600h10m44.851661896s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection Dec 30 15:39:44.980584171 (600h10m44.851667206s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold Dec 30 15:39:44.980730901 (600h10m44.851813936s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 30 15:39:44.980747002 (600h10m44.851830037s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold Dec 30 15:39:44.980793770 (600h10m44.851876806s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127NKToES711diH4f8TFJbwERq3NhhFEFrQyi7Mka3VVLsamuzp): protocol: expected piece hash; context canceled; EOF Dec 30 15:39:44.980799651 (600h10m44.851882687s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold Dec 30 15:39:44.980930771 (600h10m44.852013806s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nnXd911ApLeiKnc4LgDEUNnnkxtnLEdwzP48BhuhyEykacV6W): protocol: expected piece hash; context canceled; EOF Dec 30 15:39:44.980937775 (600h10m44.85202081s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold Dec 30 15:39:44.980956481 (600h10m44.852039526s) [seg_pos={0 0}, upload=526030]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 15:39:44.981039398 (600h10m44.852122433s) [upload=526030]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 15:39:44.981056070 (600h10m44.852139105s) [upload=526030]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 15:39:45.290276710 (600h10m45.161359765s) [upload=526030]: Done uploading segments... Dec 30 15:39:45.290299182 (600h10m45.161382227s) [upload=526030]: Upload finished. err: <nil> Duration: 45.231158328s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 11.90912863s trace: https://jaeger.dev.storj.io/trace/547648a2cfce50e7 FAIL (2025-12-30 15:53:04 UTC) orbiter warning: context deadline exceeded Duration: 11.909157375s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X.......X.X..........XX.......XX.XX..X....X.......X......X.] duration: 6.230676767s trace: https://jaeger.dev.storj.io/trace/77d7b07556375ae9 FAIL (2025-12-30 15:52:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 15:52:21.521824104 (600h23m21.392907139s) [upload=526214]: Starting upload Dec 30 15:52:21.521919696 (600h23m21.393002751s) [upload=526214]: Done starting upload Dec 30 15:52:21.522021430 (600h23m21.393104495s) [upload=526214]: Uploading segments... Dec 30 15:52:21.522190452 (600h23m21.393273497s) [upload=526214]: Got next segment. Inline: false Dec 30 15:52:21.522251758 (600h23m21.393334803s) [upload=526214]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 15:52:21.522258961 (600h23m21.393342006s) [upload=526214]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 15:52:21.768362000 (600h23m21.639445035s) [seg_pos={0 0}, upload=526214]: Begin upload segment... Dec 30 15:52:21.768402778 (600h23m21.639485823s) [seg_pos={0 0}, upload=526214]: Done begin upload segment. Dec 30 15:52:21.768429549 (600h23m21.639512584s) [noise=true , address=145.79.227.10:4403 , piece_id=F3VN3XJKUY3A5QQSA2YDKQMYIIKPC6OUYXG6ZX3SDBJZOHEUGX4A, seg_pos={0 0}, upload=526214]: Uploading piece... Dec 30 15:52:21.768461470 (600h23m21.639544505s) [noise=true , address=145.79.227.20:4438 , piece_id=YZEVD4I3VCXI6EOPBCA3VMI7YJT6GX6RDO37LE2ZC22TFN2PAQDQ, seg_pos={0 0}, upload=526214]: Uploading piece... Dec 30 15:52:21.768936374 (600h23m21.640019419s) [noise=true , address=145.79.227.39:4013 , piece_id=5P4MPUNKJZNKAPRDBGWTYOOKEQ5QP45AUBTMZZAMXCIT7DFWI75Q, seg_pos={0 0}, upload=526214]: Uploading piece... Dec 30 15:52:21.769336576 (600h23m21.640419611s) [noise=true , address=145.79.227.24:4582 , piece_id=NQZLU2D5LKSDZLUCRJK4QTJFRJDN3ECTS3XOLMEBIWWDDYTY3WVQ, seg_pos={0 0}, upload=526214]: Uploading piece... Dec 30 15:52:21.769752127 (600h23m21.640835173s) [noise=true , address=45.91.116.5:4297 , piece_id=UCBKRF2JJMPYHQ76N2APP2Y5VSEESMP6Z67KDQDDK7XOKOFGBUBA, seg_pos={0 0}, upload=526214]: Uploading piece... Dec 30 15:52:21.770240768 (600h23m21.641323803s) [noise=true , address=145.79.227.35:4482 , piece_id=ULG5ULWDGUEYVOQN2TSGOWMBVIVX2ID2WY52ID2VTZZ2DN4N7ZOA, seg_pos={0 0}, upload=526214]: Uploading piece... Dec 30 15:52:21.770692167 (600h23m21.641775202s) [noise=true , address=45.91.116.11:4118 , piece_id=UNSQ4VDAMOTILYBTDD3JL7HEGPWZTIWJ3XOG7565WHZX4G75JBOQ, seg_pos={0 0}, upload=526214]: Uploading piece... Dec 30 15:52:21.771916749 (600h23m21.642999784s) [noise=true , address=45.91.116.14:4190 , piece_id=SXCB2HSHXHHNQAU6WOIQRGFAD2ZN5BKGN7X3Y4ZMEIOWSCOSGXQA, seg_pos={0 0}, upload=526214]: Uploading piece... Dec 30 15:52:21.772344634 (600h23m21.643427679s) [noise=true , address=45.91.116.12:4225 , piece_id=7HLOUA6OZ4WWT4T5XS6ALU22Y6ABGNXHD4KTMVHGNHNMTVH6LRBQ, seg_pos={0 0}, upload=526214]: Uploading piece... Dec 30 15:52:21.773065266 (600h23m21.644148301s) [noise=true , address=145.79.227.4:4185 , piece_id=5RXLGY4PU4DPN7KPVHW7WVKUAFVT5BZ52RLUBT7VSPDOYYRGVAWA, seg_pos={0 0}, upload=526214]: Uploading piece... Dec 30 15:52:21.774161974 (600h23m21.645245019s) [noise=true , address=145.79.227.16:4912 , piece_id=S5X7S73PI2RTHEAIFQRURMIZUENVPNLCGEJOVQ5Y3JNFJGSN53GA, seg_pos={0 0}, upload=526214]: Uploading piece... Dec 30 15:52:21.774194396 (600h23m21.645277441s) [noise=true , address=45.91.116.7:4012 , piece_id=4WUASTAQP6QP4PMPRLJ5KZUIO5RYIL46YWSMHLITI2TRNGIHFS4A, seg_pos={0 0}, upload=526214]: Uploading piece... Dec 30 15:52:21.775412255 (600h23m21.64649529s) [noise=true , address=145.79.227.47:4267 , piece_id=QNDALOBDFNREC37B6X566YEHJDG6VEOROCVDF3NJNEXGRCM4N45Q, seg_pos={0 0}, upload=526214]: Uploading piece... Dec 30 15:52:21.776486480 (600h23m21.647569515s) [noise=true , address=145.79.227.50:4373 , piece_id=S7JUBAAWR7MI35JQABEQWLXOWNOQDS4MLKSDCXQC2SS7FFT3FXAQ, seg_pos={0 0}, upload=526214]: Uploading piece... Dec 30 15:52:21.777571206 (600h23m21.648654241s) [noise=true , address=145.79.227.14:4840 , piece_id=UMEC5D6N6VYZMU266AFGHLKXLXYBWEE2TKMZZYE3I6W2QTXNHVOQ, seg_pos={0 0}, upload=526214]: Uploading piece... Dec 30 15:52:21.778756623 (600h23m21.649839668s) [noise=true , address=77.90.31.10:4517 , piece_id=JHR3TDHKJEQ7VU6BI2BVD6OE5FDCKGFKIOWHRGBMAXRRB5U2YTWQ, seg_pos={0 0}, upload=526214]: Uploading piece... Dec 30 15:52:23.482153465 (600h23m23.353236501s) [upload=526214]: Next returned nil segment Dec 30 15:52:23.482166741 (600h23m23.353249776s) [upload=526214]: Waiting for error group managing segments... Dec 30 15:52:23.758058321 (600h23m23.629141356s) [noise=true , address=145.79.227.10:4403 , piece_id=F3VN3XJKUY3A5QQSA2YDKQMYIIKPC6OUYXG6ZX3SDBJZOHEUGX4A, seg_pos={0 0}, upload=526214]: Done uploading piece. err: <nil> Dec 30 15:52:23.758248292 (600h23m23.629331337s) [noise=true , address=145.79.227.4:4185 , piece_id=5RXLGY4PU4DPN7KPVHW7WVKUAFVT5BZ52RLUBT7VSPDOYYRGVAWA, seg_pos={0 0}, upload=526214]: Done uploading piece. err: <nil> Dec 30 15:52:23.763900195 (600h23m23.63498323s) [noise=true , address=145.79.227.50:4373 , piece_id=S7JUBAAWR7MI35JQABEQWLXOWNOQDS4MLKSDCXQC2SS7FFT3FXAQ, seg_pos={0 0}, upload=526214]: Done uploading piece. err: <nil> Dec 30 15:52:23.768113999 (600h23m23.639197044s) [noise=true , address=145.79.227.39:4013 , piece_id=5P4MPUNKJZNKAPRDBGWTYOOKEQ5QP45AUBTMZZAMXCIT7DFWI75Q, seg_pos={0 0}, upload=526214]: Done uploading piece. err: <nil> Dec 30 15:52:23.774452579 (600h23m23.645535624s) [noise=true , address=145.79.227.47:4267 , piece_id=QNDALOBDFNREC37B6X566YEHJDG6VEOROCVDF3NJNEXGRCM4N45Q, seg_pos={0 0}, upload=526214]: Done uploading piece. err: <nil> Dec 30 15:52:23.780886901 (600h23m23.651969936s) [noise=true , address=145.79.227.16:4912 , piece_id=S5X7S73PI2RTHEAIFQRURMIZUENVPNLCGEJOVQ5Y3JNFJGSN53GA, seg_pos={0 0}, upload=526214]: Done uploading piece. err: <nil> Dec 30 15:52:23.783325464 (600h23m23.654408499s) [noise=true , address=145.79.227.20:4438 , piece_id=YZEVD4I3VCXI6EOPBCA3VMI7YJT6GX6RDO37LE2ZC22TFN2PAQDQ, seg_pos={0 0}, upload=526214]: Done uploading piece. err: <nil> Dec 30 15:52:23.848650803 (600h23m23.719733838s) [noise=true , address=145.79.227.24:4582 , piece_id=NQZLU2D5LKSDZLUCRJK4QTJFRJDN3ECTS3XOLMEBIWWDDYTY3WVQ, seg_pos={0 0}, upload=526214]: Done uploading piece. err: <nil> Dec 30 15:52:24.093942757 (600h23m23.965025792s) [noise=true , address=45.91.116.14:4190 , piece_id=SXCB2HSHXHHNQAU6WOIQRGFAD2ZN5BKGN7X3Y4ZMEIOWSCOSGXQA, seg_pos={0 0}, upload=526214]: Done uploading piece. err: <nil> Dec 30 15:52:24.169506513 (600h23m24.040589558s) [noise=true , address=45.91.116.11:4118 , piece_id=UNSQ4VDAMOTILYBTDD3JL7HEGPWZTIWJ3XOG7565WHZX4G75JBOQ, seg_pos={0 0}, upload=526214]: Done uploading piece. err: <nil> Dec 30 15:52:24.312602400 (600h23m24.183685435s) [noise=true , address=45.91.116.5:4297 , piece_id=UCBKRF2JJMPYHQ76N2APP2Y5VSEESMP6Z67KDQDDK7XOKOFGBUBA, seg_pos={0 0}, upload=526214]: Done uploading piece. err: <nil> Dec 30 15:52:24.561583056 (600h23m24.432666091s) [noise=true , address=145.79.227.14:4840 , piece_id=UMEC5D6N6VYZMU266AFGHLKXLXYBWEE2TKMZZYE3I6W2QTXNHVOQ, seg_pos={0 0}, upload=526214]: Done uploading piece. err: <nil> Dec 30 15:52:24.592530315 (600h23m24.46361335s) [noise=true , address=77.90.31.10:4517 , piece_id=JHR3TDHKJEQ7VU6BI2BVD6OE5FDCKGFKIOWHRGBMAXRRB5U2YTWQ, seg_pos={0 0}, upload=526214]: Done uploading piece. err: <nil> Dec 30 15:52:24.592550423 (600h23m24.463633458s) [seg_pos={0 0}, upload=526214]: Segment reached optimal threshold of 13 pieces. Dec 30 15:52:24.592745214 (600h23m24.463828249s) [noise=true , address=45.91.116.7:4012 , piece_id=4WUASTAQP6QP4PMPRLJ5KZUIO5RYIL46YWSMHLITI2TRNGIHFS4A, seg_pos={0 0}, upload=526214]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 30 15:52:24.592753309 (600h23m24.463836354s) [noise=true , address=45.91.116.7:4012 , piece_id=4WUASTAQP6QP4PMPRLJ5KZUIO5RYIL46YWSMHLITI2TRNGIHFS4A, seg_pos={0 0}, upload=526214]: Long tail cancellation due to optimal threshold Dec 30 15:52:24.592927812 (600h23m24.464010857s) [noise=true , address=45.91.116.12:4225 , piece_id=7HLOUA6OZ4WWT4T5XS6ALU22Y6ABGNXHD4KTMVHGNHNMTVH6LRBQ, seg_pos={0 0}, upload=526214]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 30 15:52:24.592933522 (600h23m24.464016567s) [noise=true , address=45.91.116.12:4225 , piece_id=7HLOUA6OZ4WWT4T5XS6ALU22Y6ABGNXHD4KTMVHGNHNMTVH6LRBQ, seg_pos={0 0}, upload=526214]: Long tail cancellation due to optimal threshold Dec 30 15:52:24.592995921 (600h23m24.464078967s) [noise=true , address=145.79.227.35:4482 , piece_id=ULG5ULWDGUEYVOQN2TSGOWMBVIVX2ID2WY52ID2VTZZ2DN4N7ZOA, seg_pos={0 0}, upload=526214]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 30 15:52:24.593010981 (600h23m24.464094016s) [noise=true , address=145.79.227.35:4482 , piece_id=ULG5ULWDGUEYVOQN2TSGOWMBVIVX2ID2WY52ID2VTZZ2DN4N7ZOA, seg_pos={0 0}, upload=526214]: Long tail cancellation due to optimal threshold Dec 30 15:52:24.593069632 (600h23m24.464152677s) [seg_pos={0 0}, upload=526214]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 15:52:24.593182847 (600h23m24.464265882s) [upload=526214]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 15:52:24.593188479 (600h23m24.464271524s) [upload=526214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 15:52:24.892592971 (600h23m24.763676007s) [upload=526214]: Done uploading segments... Dec 30 15:52:24.892613149 (600h23m24.763696184s) [upload=526214]: Upload finished. err: <nil> Duration: 6.230693558s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..X.......X.X..........XX.......XX.XX..X....X.......X......X] duration: 30.002933011s trace: https://jaeger.dev.storj.io/trace/77d7b07556375ae9 FAIL (2025-12-30 15:52:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 30 15:52:21.521824104 (600h23m21.392907139s) [upload=526214]: Starting upload Dec 30 15:52:21.521919696 (600h23m21.393002751s) [upload=526214]: Done starting upload Dec 30 15:52:21.522021430 (600h23m21.393104495s) [upload=526214]: Uploading segments... Dec 30 15:52:21.522190452 (600h23m21.393273497s) [upload=526214]: Got next segment. Inline: false Dec 30 15:52:21.522251758 (600h23m21.393334803s) [upload=526214]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 15:52:21.522258961 (600h23m21.393342006s) [upload=526214]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 15:52:21.768362000 (600h23m21.639445035s) [seg_pos={0 0}, upload=526214]: Begin upload segment... Dec 30 15:52:21.768402778 (600h23m21.639485823s) [seg_pos={0 0}, upload=526214]: Done begin upload segment. Dec 30 15:52:21.768429549 (600h23m21.639512584s) [noise=true , address=145.79.227.10:4403 , piece_id=F3VN3XJKUY3A5QQSA2YDKQMYIIKPC6OUYXG6ZX3SDBJZOHEUGX4A, seg_pos={0 0}, upload=526214]: Uploading piece... Dec 30 15:52:21.768461470 (600h23m21.639544505s) [noise=true , address=145.79.227.20:4438 , piece_id=YZEVD4I3VCXI6EOPBCA3VMI7YJT6GX6RDO37LE2ZC22TFN2PAQDQ, seg_pos={0 0}, upload=526214]: Uploading piece... Dec 30 15:52:21.768936374 (600h23m21.640019419s) [noise=true , address=145.79.227.39:4013 , piece_id=5P4MPUNKJZNKAPRDBGWTYOOKEQ5QP45AUBTMZZAMXCIT7DFWI75Q, seg_pos={0 0}, upload=526214]: Uploading piece... Dec 30 15:52:21.769336576 (600h23m21.640419611s) [noise=true , address=145.79.227.24:4582 , piece_id=NQZLU2D5LKSDZLUCRJK4QTJFRJDN3ECTS3XOLMEBIWWDDYTY3WVQ, seg_pos={0 0}, upload=526214]: Uploading piece... Dec 30 15:52:21.769752127 (600h23m21.640835173s) [noise=true , address=45.91.116.5:4297 , piece_id=UCBKRF2JJMPYHQ76N2APP2Y5VSEESMP6Z67KDQDDK7XOKOFGBUBA, seg_pos={0 0}, upload=526214]: Uploading piece... Dec 30 15:52:21.770240768 (600h23m21.641323803s) [noise=true , address=145.79.227.35:4482 , piece_id=ULG5ULWDGUEYVOQN2TSGOWMBVIVX2ID2WY52ID2VTZZ2DN4N7ZOA, seg_pos={0 0}, upload=526214]: Uploading piece... Dec 30 15:52:21.770692167 (600h23m21.641775202s) [noise=true , address=45.91.116.11:4118 , piece_id=UNSQ4VDAMOTILYBTDD3JL7HEGPWZTIWJ3XOG7565WHZX4G75JBOQ, seg_pos={0 0}, upload=526214]: Uploading piece... Dec 30 15:52:21.771916749 (600h23m21.642999784s) [noise=true , address=45.91.116.14:4190 , piece_id=SXCB2HSHXHHNQAU6WOIQRGFAD2ZN5BKGN7X3Y4ZMEIOWSCOSGXQA, seg_pos={0 0}, upload=526214]: Uploading piece... Dec 30 15:52:21.772344634 (600h23m21.643427679s) [noise=true , address=45.91.116.12:4225 , piece_id=7HLOUA6OZ4WWT4T5XS6ALU22Y6ABGNXHD4KTMVHGNHNMTVH6LRBQ, seg_pos={0 0}, upload=526214]: Uploading piece... Dec 30 15:52:21.773065266 (600h23m21.644148301s) [noise=true , address=145.79.227.4:4185 , piece_id=5RXLGY4PU4DPN7KPVHW7WVKUAFVT5BZ52RLUBT7VSPDOYYRGVAWA, seg_pos={0 0}, upload=526214]: Uploading piece... Dec 30 15:52:21.774161974 (600h23m21.645245019s) [noise=true , address=145.79.227.16:4912 , piece_id=S5X7S73PI2RTHEAIFQRURMIZUENVPNLCGEJOVQ5Y3JNFJGSN53GA, seg_pos={0 0}, upload=526214]: Uploading piece... Dec 30 15:52:21.774194396 (600h23m21.645277441s) [noise=true , address=45.91.116.7:4012 , piece_id=4WUASTAQP6QP4PMPRLJ5KZUIO5RYIL46YWSMHLITI2TRNGIHFS4A, seg_pos={0 0}, upload=526214]: Uploading piece... Dec 30 15:52:21.775412255 (600h23m21.64649529s) [noise=true , address=145.79.227.47:4267 , piece_id=QNDALOBDFNREC37B6X566YEHJDG6VEOROCVDF3NJNEXGRCM4N45Q, seg_pos={0 0}, upload=526214]: Uploading piece... Dec 30 15:52:21.776486480 (600h23m21.647569515s) [noise=true , address=145.79.227.50:4373 , piece_id=S7JUBAAWR7MI35JQABEQWLXOWNOQDS4MLKSDCXQC2SS7FFT3FXAQ, seg_pos={0 0}, upload=526214]: Uploading piece... Dec 30 15:52:21.777571206 (600h23m21.648654241s) [noise=true , address=145.79.227.14:4840 , piece_id=UMEC5D6N6VYZMU266AFGHLKXLXYBWEE2TKMZZYE3I6W2QTXNHVOQ, seg_pos={0 0}, upload=526214]: Uploading piece... Dec 30 15:52:21.778756623 (600h23m21.649839668s) [noise=true , address=77.90.31.10:4517 , piece_id=JHR3TDHKJEQ7VU6BI2BVD6OE5FDCKGFKIOWHRGBMAXRRB5U2YTWQ, seg_pos={0 0}, upload=526214]: Uploading piece... Dec 30 15:52:23.482153465 (600h23m23.353236501s) [upload=526214]: Next returned nil segment Dec 30 15:52:23.482166741 (600h23m23.353249776s) [upload=526214]: Waiting for error group managing segments... Dec 30 15:52:23.758058321 (600h23m23.629141356s) [noise=true , address=145.79.227.10:4403 , piece_id=F3VN3XJKUY3A5QQSA2YDKQMYIIKPC6OUYXG6ZX3SDBJZOHEUGX4A, seg_pos={0 0}, upload=526214]: Done uploading piece. err: <nil> Dec 30 15:52:23.758248292 (600h23m23.629331337s) [noise=true , address=145.79.227.4:4185 , piece_id=5RXLGY4PU4DPN7KPVHW7WVKUAFVT5BZ52RLUBT7VSPDOYYRGVAWA, seg_pos={0 0}, upload=526214]: Done uploading piece. err: <nil> Dec 30 15:52:23.763900195 (600h23m23.63498323s) [noise=true , address=145.79.227.50:4373 , piece_id=S7JUBAAWR7MI35JQABEQWLXOWNOQDS4MLKSDCXQC2SS7FFT3FXAQ, seg_pos={0 0}, upload=526214]: Done uploading piece. err: <nil> Dec 30 15:52:23.768113999 (600h23m23.639197044s) [noise=true , address=145.79.227.39:4013 , piece_id=5P4MPUNKJZNKAPRDBGWTYOOKEQ5QP45AUBTMZZAMXCIT7DFWI75Q, seg_pos={0 0}, upload=526214]: Done uploading piece. err: <nil> Dec 30 15:52:23.774452579 (600h23m23.645535624s) [noise=true , address=145.79.227.47:4267 , piece_id=QNDALOBDFNREC37B6X566YEHJDG6VEOROCVDF3NJNEXGRCM4N45Q, seg_pos={0 0}, upload=526214]: Done uploading piece. err: <nil> Dec 30 15:52:23.780886901 (600h23m23.651969936s) [noise=true , address=145.79.227.16:4912 , piece_id=S5X7S73PI2RTHEAIFQRURMIZUENVPNLCGEJOVQ5Y3JNFJGSN53GA, seg_pos={0 0}, upload=526214]: Done uploading piece. err: <nil> Dec 30 15:52:23.783325464 (600h23m23.654408499s) [noise=true , address=145.79.227.20:4438 , piece_id=YZEVD4I3VCXI6EOPBCA3VMI7YJT6GX6RDO37LE2ZC22TFN2PAQDQ, seg_pos={0 0}, upload=526214]: Done uploading piece. err: <nil> Dec 30 15:52:23.848650803 (600h23m23.719733838s) [noise=true , address=145.79.227.24:4582 , piece_id=NQZLU2D5LKSDZLUCRJK4QTJFRJDN3ECTS3XOLMEBIWWDDYTY3WVQ, seg_pos={0 0}, upload=526214]: Done uploading piece. err: <nil> Dec 30 15:52:24.093942757 (600h23m23.965025792s) [noise=true , address=45.91.116.14:4190 , piece_id=SXCB2HSHXHHNQAU6WOIQRGFAD2ZN5BKGN7X3Y4ZMEIOWSCOSGXQA, seg_pos={0 0}, upload=526214]: Done uploading piece. err: <nil> Dec 30 15:52:24.169506513 (600h23m24.040589558s) [noise=true , address=45.91.116.11:4118 , piece_id=UNSQ4VDAMOTILYBTDD3JL7HEGPWZTIWJ3XOG7565WHZX4G75JBOQ, seg_pos={0 0}, upload=526214]: Done uploading piece. err: <nil> Dec 30 15:52:24.312602400 (600h23m24.183685435s) [noise=true , address=45.91.116.5:4297 , piece_id=UCBKRF2JJMPYHQ76N2APP2Y5VSEESMP6Z67KDQDDK7XOKOFGBUBA, seg_pos={0 0}, upload=526214]: Done uploading piece. err: <nil> Dec 30 15:52:24.561583056 (600h23m24.432666091s) [noise=true , address=145.79.227.14:4840 , piece_id=UMEC5D6N6VYZMU266AFGHLKXLXYBWEE2TKMZZYE3I6W2QTXNHVOQ, seg_pos={0 0}, upload=526214]: Done uploading piece. err: <nil> Dec 30 15:52:24.592530315 (600h23m24.46361335s) [noise=true , address=77.90.31.10:4517 , piece_id=JHR3TDHKJEQ7VU6BI2BVD6OE5FDCKGFKIOWHRGBMAXRRB5U2YTWQ, seg_pos={0 0}, upload=526214]: Done uploading piece. err: <nil> Dec 30 15:52:24.592550423 (600h23m24.463633458s) [seg_pos={0 0}, upload=526214]: Segment reached optimal threshold of 13 pieces. Dec 30 15:52:24.592745214 (600h23m24.463828249s) [noise=true , address=45.91.116.7:4012 , piece_id=4WUASTAQP6QP4PMPRLJ5KZUIO5RYIL46YWSMHLITI2TRNGIHFS4A, seg_pos={0 0}, upload=526214]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t): protocol: expected piece hash; context canceled; EOF Dec 30 15:52:24.592753309 (600h23m24.463836354s) [noise=true , address=45.91.116.7:4012 , piece_id=4WUASTAQP6QP4PMPRLJ5KZUIO5RYIL46YWSMHLITI2TRNGIHFS4A, seg_pos={0 0}, upload=526214]: Long tail cancellation due to optimal threshold Dec 30 15:52:24.592927812 (600h23m24.464010857s) [noise=true , address=45.91.116.12:4225 , piece_id=7HLOUA6OZ4WWT4T5XS6ALU22Y6ABGNXHD4KTMVHGNHNMTVH6LRBQ, seg_pos={0 0}, upload=526214]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 30 15:52:24.592933522 (600h23m24.464016567s) [noise=true , address=45.91.116.12:4225 , piece_id=7HLOUA6OZ4WWT4T5XS6ALU22Y6ABGNXHD4KTMVHGNHNMTVH6LRBQ, seg_pos={0 0}, upload=526214]: Long tail cancellation due to optimal threshold Dec 30 15:52:24.592995921 (600h23m24.464078967s) [noise=true , address=145.79.227.35:4482 , piece_id=ULG5ULWDGUEYVOQN2TSGOWMBVIVX2ID2WY52ID2VTZZ2DN4N7ZOA, seg_pos={0 0}, upload=526214]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12mNCKAdMzWu87fmyrcwDKZvscMcu2CogjXicwAi5wLSFJa7oVV): protocol: expected piece hash; context canceled; EOF Dec 30 15:52:24.593010981 (600h23m24.464094016s) [noise=true , address=145.79.227.35:4482 , piece_id=ULG5ULWDGUEYVOQN2TSGOWMBVIVX2ID2WY52ID2VTZZ2DN4N7ZOA, seg_pos={0 0}, upload=526214]: Long tail cancellation due to optimal threshold Dec 30 15:52:24.593069632 (600h23m24.464152677s) [seg_pos={0 0}, upload=526214]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 15:52:24.593182847 (600h23m24.464265882s) [upload=526214]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 15:52:24.593188479 (600h23m24.464271524s) [upload=526214]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 15:52:24.892592971 (600h23m24.763676007s) [upload=526214]: Done uploading segments... Dec 30 15:52:24.892613149 (600h23m24.763696184s) [upload=526214]: Upload finished. err: <nil> Duration: 30.00295352s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 43 out of 60 is failed (threshold=6) [XXXXX..XXXX.XX.X.X.X.XXXXXX.XXXX...XX.X.XXXXXXXXX.XXXXXXX...] duration: 5.001211283s trace: https://jaeger.dev.storj.io/trace/2d8479babd245c3e FAIL (2025-12-30 15:52:43 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001262719s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 11.908528747s trace: https://jaeger.dev.storj.io/trace/547648a2cfce50e7 FAIL (2025-12-30 15:52:33 UTC) orbiter warning: context deadline exceeded Duration: 11.908571648s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [........X............................X..XX..X...............] duration: 39.075019397s trace: https://jaeger.dev.storj.io/trace/3b27a789ef618091 FAIL (2025-12-30 15:52:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ise=true , address=145.79.227.5:4252 , piece_id=JRKV35AO742VMYZPN3WKPNRZKO2ZFUU53RS4MXF22HO5UC6BKCOQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.081840064 (600h10m41.952923099s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.082191573 (600h10m41.953274608s) [noise=true , address=145.79.227.16:4942 , piece_id=G5I53KG36VHVW5LICAHOHRXTQVQ67ZFGJQNXRJMYNOIT7WB7XTEQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.082429897 (600h10m41.953512932s) [noise=true , address=145.79.227.29:4968 , piece_id=PMTNKRHI6YWMJPZ2R7RPIF7Q6CFLFKBH5OTHWCX4RNGIAFURVMOQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.082680584 (600h10m41.953763619s) [noise=true , address=145.79.227.33:4892 , piece_id=RSEJC3XHFACGLV6YBHYM7QBCU7MMYST53FJO4ITLA2C3IHBZCRTQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.082917976 (600h10m41.954001011s) [noise=true , address=77.90.31.12:4596 , piece_id=TQVNHTTAWVUCUIHSHWWH27MFJNOE2MWLJCRMTPUCREML7DT3RUYQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.083149276 (600h10m41.954232311s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.083393431 (600h10m41.954476466s) [noise=true , address=145.79.227.35:4508 , piece_id=KJJFNTPYUUHLFFCGHD3KDTN3DK6C2R5YRLWJDMF6C4UGSMFYVFBA, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.083842486 (600h10m41.954925521s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.086214362 (600h10m41.957297397s) [noise=true , address=145.79.227.49:4367 , piece_id=QUI5FSOTJVIHT67I3F2YDB5IPKSYEYJTDRKA44UM355S5FSNNLUA, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.086612852 (600h10m41.957695887s) [noise=true , address=77.90.31.13:4652 , piece_id=5VR2F6XUEYGX5KULBIL6V3YGH7Q3T5UYFJTS5RGWN3B7F7TVMHEQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.087061346 (600h10m41.958144381s) [noise=true , address=77.90.31.11:4588 , piece_id=WY5XS4EPFHDLCT3XEQQKBO2M5GLVTBNDZOKPTFPVZV6EALPMNSHA, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.087452310 (600h10m41.958535345s) [noise=true , address=145.79.227.30:4768 , piece_id=2KZKXIYGSDES5Q5W2PVAACJYU6ANKQNDP2VQEBXM7QKXOT6QLSBA, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.087903780 (600h10m41.958986825s) [noise=true , address=64.49.13.27:4957 , piece_id=ETFKCNNSFAVJWNB3J3U4LPRDGBILNNA5CK7FR7X5UHONPZHXFBLQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.088521366 (600h10m41.959604411s) [noise=true , address=193.149.164.143:4698 , piece_id=F6B3UPOKCSD7ZXQ4SYKOAALEBA4FSTQ2ELGMGCU7TMWVQZDZB2SQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.088618600 (600h10m41.959701645s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:43.511091675 (600h10m43.38217472s) [upload=526030]: Next returned nil segment Dec 30 15:39:43.511109379 (600h10m43.382192424s) [upload=526030]: Waiting for error group managing segments... Dec 30 15:39:44.017333248 (600h10m43.888416283s) [noise=true , address=145.79.227.5:4252 , piece_id=JRKV35AO742VMYZPN3WKPNRZKO2ZFUU53RS4MXF22HO5UC6BKCOQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.069300150 (600h10m43.940383186s) [noise=true , address=64.49.13.27:4957 , piece_id=ETFKCNNSFAVJWNB3J3U4LPRDGBILNNA5CK7FR7X5UHONPZHXFBLQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.126868228 (600h10m43.997951273s) [noise=true , address=77.90.31.17:4800 , piece_id=PYZVLAZZKWP6UOL6HJVVWSEJOXCJWK5IAMWJPRODOKCLRZE37BAQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.146896235 (600h10m44.01797927s) [noise=true , address=145.79.227.29:4968 , piece_id=PMTNKRHI6YWMJPZ2R7RPIF7Q6CFLFKBH5OTHWCX4RNGIAFURVMOQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.166221683 (600h10m44.037304718s) [noise=true , address=77.90.31.12:4596 , piece_id=TQVNHTTAWVUCUIHSHWWH27MFJNOE2MWLJCRMTPUCREML7DT3RUYQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.234530763 (600h10m44.105613808s) [noise=true , address=77.90.31.13:4652 , piece_id=5VR2F6XUEYGX5KULBIL6V3YGH7Q3T5UYFJTS5RGWN3B7F7TVMHEQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.489468910 (600h10m44.360551955s) [noise=true , address=145.79.227.16:4942 , piece_id=G5I53KG36VHVW5LICAHOHRXTQVQ67ZFGJQNXRJMYNOIT7WB7XTEQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.802041331 (600h10m44.673124366s) [noise=true , address=193.149.164.143:4698 , piece_id=F6B3UPOKCSD7ZXQ4SYKOAALEBA4FSTQ2ELGMGCU7TMWVQZDZB2SQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.822093824 (600h10m44.693176859s) [noise=true , address=145.79.227.49:4367 , piece_id=QUI5FSOTJVIHT67I3F2YDB5IPKSYEYJTDRKA44UM355S5FSNNLUA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.867276618 (600h10m44.738359653s) [noise=true , address=145.79.227.30:4768 , piece_id=2KZKXIYGSDES5Q5W2PVAACJYU6ANKQNDP2VQEBXM7QKXOT6QLSBA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.881025168 (600h10m44.752108203s) [noise=true , address=145.79.227.44:4199 , piece_id=AJKADWXHHMCPE6V52UMJJPCHOBBL7QDU7DF337LB6Q3YDHEYUUQQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.899437537 (600h10m44.770520572s) [noise=true , address=77.90.31.11:4588 , piece_id=WY5XS4EPFHDLCT3XEQQKBO2M5GLVTBNDZOKPTFPVZV6EALPMNSHA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.941939747 (600h10m44.813022782s) [noise=true , address=145.79.227.33:4892 , piece_id=RSEJC3XHFACGLV6YBHYM7QBCU7MMYST53FJO4ITLA2C3IHBZCRTQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.980395983 (600h10m44.851479029s) [noise=true , address=145.79.227.35:4508 , piece_id=KJJFNTPYUUHLFFCGHD3KDTN3DK6C2R5YRLWJDMF6C4UGSMFYVFBA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.980415871 (600h10m44.851498916s) [seg_pos={0 0}, upload=526030]: Segment reached optimal threshold of 14 pieces. Dec 30 15:39:44.980578861 (600h10m44.851661896s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection Dec 30 15:39:44.980584171 (600h10m44.851667206s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold Dec 30 15:39:44.980730901 (600h10m44.851813936s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 30 15:39:44.980747002 (600h10m44.851830037s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold Dec 30 15:39:44.980793770 (600h10m44.851876806s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127NKToES711diH4f8TFJbwERq3NhhFEFrQyi7Mka3VVLsamuzp): protocol: expected piece hash; context canceled; EOF Dec 30 15:39:44.980799651 (600h10m44.851882687s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold Dec 30 15:39:44.980930771 (600h10m44.852013806s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nnXd911ApLeiKnc4LgDEUNnnkxtnLEdwzP48BhuhyEykacV6W): protocol: expected piece hash; context canceled; EOF Dec 30 15:39:44.980937775 (600h10m44.85202081s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold Dec 30 15:39:44.980956481 (600h10m44.852039526s) [seg_pos={0 0}, upload=526030]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 15:39:44.981039398 (600h10m44.852122433s) [upload=526030]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 15:39:44.981056070 (600h10m44.852139105s) [upload=526030]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 15:39:45.290276710 (600h10m45.161359765s) [upload=526030]: Done uploading segments... Dec 30 15:39:45.290299182 (600h10m45.161382227s) [upload=526030]: Upload finished. err: <nil> Duration: 39.075033784s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 13.901941617s trace: https://jaeger.dev.storj.io/trace/547648a2cfce50e7 FAIL (2025-12-30 15:51:59 UTC) orbiter warning: context deadline exceeded Duration: 13.901966675s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........X............................X..XX..X..............] duration: 41.698860392s trace: https://jaeger.dev.storj.io/trace/3b27a789ef618091 FAIL (2025-12-30 15:51:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ise=true , address=145.79.227.5:4252 , piece_id=JRKV35AO742VMYZPN3WKPNRZKO2ZFUU53RS4MXF22HO5UC6BKCOQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.081840064 (600h10m41.952923099s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.082191573 (600h10m41.953274608s) [noise=true , address=145.79.227.16:4942 , piece_id=G5I53KG36VHVW5LICAHOHRXTQVQ67ZFGJQNXRJMYNOIT7WB7XTEQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.082429897 (600h10m41.953512932s) [noise=true , address=145.79.227.29:4968 , piece_id=PMTNKRHI6YWMJPZ2R7RPIF7Q6CFLFKBH5OTHWCX4RNGIAFURVMOQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.082680584 (600h10m41.953763619s) [noise=true , address=145.79.227.33:4892 , piece_id=RSEJC3XHFACGLV6YBHYM7QBCU7MMYST53FJO4ITLA2C3IHBZCRTQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.082917976 (600h10m41.954001011s) [noise=true , address=77.90.31.12:4596 , piece_id=TQVNHTTAWVUCUIHSHWWH27MFJNOE2MWLJCRMTPUCREML7DT3RUYQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.083149276 (600h10m41.954232311s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.083393431 (600h10m41.954476466s) [noise=true , address=145.79.227.35:4508 , piece_id=KJJFNTPYUUHLFFCGHD3KDTN3DK6C2R5YRLWJDMF6C4UGSMFYVFBA, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.083842486 (600h10m41.954925521s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.086214362 (600h10m41.957297397s) [noise=true , address=145.79.227.49:4367 , piece_id=QUI5FSOTJVIHT67I3F2YDB5IPKSYEYJTDRKA44UM355S5FSNNLUA, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.086612852 (600h10m41.957695887s) [noise=true , address=77.90.31.13:4652 , piece_id=5VR2F6XUEYGX5KULBIL6V3YGH7Q3T5UYFJTS5RGWN3B7F7TVMHEQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.087061346 (600h10m41.958144381s) [noise=true , address=77.90.31.11:4588 , piece_id=WY5XS4EPFHDLCT3XEQQKBO2M5GLVTBNDZOKPTFPVZV6EALPMNSHA, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.087452310 (600h10m41.958535345s) [noise=true , address=145.79.227.30:4768 , piece_id=2KZKXIYGSDES5Q5W2PVAACJYU6ANKQNDP2VQEBXM7QKXOT6QLSBA, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.087903780 (600h10m41.958986825s) [noise=true , address=64.49.13.27:4957 , piece_id=ETFKCNNSFAVJWNB3J3U4LPRDGBILNNA5CK7FR7X5UHONPZHXFBLQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.088521366 (600h10m41.959604411s) [noise=true , address=193.149.164.143:4698 , piece_id=F6B3UPOKCSD7ZXQ4SYKOAALEBA4FSTQ2ELGMGCU7TMWVQZDZB2SQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.088618600 (600h10m41.959701645s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:43.511091675 (600h10m43.38217472s) [upload=526030]: Next returned nil segment Dec 30 15:39:43.511109379 (600h10m43.382192424s) [upload=526030]: Waiting for error group managing segments... Dec 30 15:39:44.017333248 (600h10m43.888416283s) [noise=true , address=145.79.227.5:4252 , piece_id=JRKV35AO742VMYZPN3WKPNRZKO2ZFUU53RS4MXF22HO5UC6BKCOQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.069300150 (600h10m43.940383186s) [noise=true , address=64.49.13.27:4957 , piece_id=ETFKCNNSFAVJWNB3J3U4LPRDGBILNNA5CK7FR7X5UHONPZHXFBLQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.126868228 (600h10m43.997951273s) [noise=true , address=77.90.31.17:4800 , piece_id=PYZVLAZZKWP6UOL6HJVVWSEJOXCJWK5IAMWJPRODOKCLRZE37BAQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.146896235 (600h10m44.01797927s) [noise=true , address=145.79.227.29:4968 , piece_id=PMTNKRHI6YWMJPZ2R7RPIF7Q6CFLFKBH5OTHWCX4RNGIAFURVMOQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.166221683 (600h10m44.037304718s) [noise=true , address=77.90.31.12:4596 , piece_id=TQVNHTTAWVUCUIHSHWWH27MFJNOE2MWLJCRMTPUCREML7DT3RUYQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.234530763 (600h10m44.105613808s) [noise=true , address=77.90.31.13:4652 , piece_id=5VR2F6XUEYGX5KULBIL6V3YGH7Q3T5UYFJTS5RGWN3B7F7TVMHEQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.489468910 (600h10m44.360551955s) [noise=true , address=145.79.227.16:4942 , piece_id=G5I53KG36VHVW5LICAHOHRXTQVQ67ZFGJQNXRJMYNOIT7WB7XTEQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.802041331 (600h10m44.673124366s) [noise=true , address=193.149.164.143:4698 , piece_id=F6B3UPOKCSD7ZXQ4SYKOAALEBA4FSTQ2ELGMGCU7TMWVQZDZB2SQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.822093824 (600h10m44.693176859s) [noise=true , address=145.79.227.49:4367 , piece_id=QUI5FSOTJVIHT67I3F2YDB5IPKSYEYJTDRKA44UM355S5FSNNLUA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.867276618 (600h10m44.738359653s) [noise=true , address=145.79.227.30:4768 , piece_id=2KZKXIYGSDES5Q5W2PVAACJYU6ANKQNDP2VQEBXM7QKXOT6QLSBA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.881025168 (600h10m44.752108203s) [noise=true , address=145.79.227.44:4199 , piece_id=AJKADWXHHMCPE6V52UMJJPCHOBBL7QDU7DF337LB6Q3YDHEYUUQQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.899437537 (600h10m44.770520572s) [noise=true , address=77.90.31.11:4588 , piece_id=WY5XS4EPFHDLCT3XEQQKBO2M5GLVTBNDZOKPTFPVZV6EALPMNSHA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.941939747 (600h10m44.813022782s) [noise=true , address=145.79.227.33:4892 , piece_id=RSEJC3XHFACGLV6YBHYM7QBCU7MMYST53FJO4ITLA2C3IHBZCRTQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.980395983 (600h10m44.851479029s) [noise=true , address=145.79.227.35:4508 , piece_id=KJJFNTPYUUHLFFCGHD3KDTN3DK6C2R5YRLWJDMF6C4UGSMFYVFBA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.980415871 (600h10m44.851498916s) [seg_pos={0 0}, upload=526030]: Segment reached optimal threshold of 14 pieces. Dec 30 15:39:44.980578861 (600h10m44.851661896s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection Dec 30 15:39:44.980584171 (600h10m44.851667206s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold Dec 30 15:39:44.980730901 (600h10m44.851813936s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 30 15:39:44.980747002 (600h10m44.851830037s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold Dec 30 15:39:44.980793770 (600h10m44.851876806s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127NKToES711diH4f8TFJbwERq3NhhFEFrQyi7Mka3VVLsamuzp): protocol: expected piece hash; context canceled; EOF Dec 30 15:39:44.980799651 (600h10m44.851882687s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold Dec 30 15:39:44.980930771 (600h10m44.852013806s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nnXd911ApLeiKnc4LgDEUNnnkxtnLEdwzP48BhuhyEykacV6W): protocol: expected piece hash; context canceled; EOF Dec 30 15:39:44.980937775 (600h10m44.85202081s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold Dec 30 15:39:44.980956481 (600h10m44.852039526s) [seg_pos={0 0}, upload=526030]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 15:39:44.981039398 (600h10m44.852122433s) [upload=526030]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 15:39:44.981056070 (600h10m44.852139105s) [upload=526030]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 15:39:45.290276710 (600h10m45.161359765s) [upload=526030]: Done uploading segments... Dec 30 15:39:45.290299182 (600h10m45.161382227s) [upload=526030]: Upload finished. err: <nil> Duration: 41.698878627s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 12.355962642s trace: https://jaeger.dev.storj.io/trace/547648a2cfce50e7 FAIL (2025-12-30 15:51:39 UTC) orbiter warning: context deadline exceeded Duration: 12.355990675s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 43 out of 60 is failed (threshold=6) [.XXXXX..XXXX.XX.X.X.X.XXXXXX.XXXX...XX.X.XXXXXXXXX.XXXXXXX..] duration: 5.001275235s trace: https://jaeger.dev.storj.io/trace/2d8479babd245c3e FAIL (2025-12-30 15:51:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001301073s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [...X.......X.X..........XX.......XX.XX..X....X.......X......] duration: 5.833313374s trace: https://jaeger.dev.storj.io/trace/161ad17247435e8e FAIL (2025-12-30 15:51:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: FFTHIKWMLVVXVH7GQJH7IRHO2MM2KVM2CIOWAOAAOF3ZD4MSKAGA) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: R7YJLOWLESSQOR5QI6EDTZPNW764K57E5P5PPTDNCERCSDR4H4MA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: IN5UH4JCQNMYNYACZH6QFMXLRTQSQC2NAIH7AMN6EETKVHBINCRQ) context deadline exceeded ======LOGS====== 30 15:47:21.962125072 (600h18m21.833208107s) [upload=526138]: Starting upload Dec 30 15:47:21.962182793 (600h18m21.833265828s) [upload=526138]: Done starting upload Dec 30 15:47:21.962460822 (600h18m21.833543857s) [upload=526138]: Uploading segments... Dec 30 15:47:21.963187436 (600h18m21.834270481s) [upload=526138]: Got next segment. Inline: false Dec 30 15:47:21.963245817 (600h18m21.834328882s) [upload=526138]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 15:47:21.963260635 (600h18m21.83434367s) [upload=526138]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 15:47:22.210595890 (600h18m22.081678935s) [seg_pos={0 0}, upload=526138]: Begin upload segment... Dec 30 15:47:22.210645103 (600h18m22.081728148s) [seg_pos={0 0}, upload=526138]: Done begin upload segment. Dec 30 15:47:22.210709887 (600h18m22.081792932s) [noise=true , address=77.90.31.11:4556 , piece_id=J27PY5SWPMLPZPCNDBIBV62G7TQPF7W7RVSW74EIKVSFGXZQV5TA, seg_pos={0 0}, upload=526138]: Uploading piece... Dec 30 15:47:22.210732190 (600h18m22.081815225s) [noise=true , address=77.90.31.14:4662 , piece_id=FFTHIKWMLVVXVH7GQJH7IRHO2MM2KVM2CIOWAOAAOF3ZD4MSKAGA, seg_pos={0 0}, upload=526138]: Uploading piece... Dec 30 15:47:22.211406824 (600h18m22.082489859s) [noise=true , address=145.79.227.51:4409 , piece_id=Y3D6WOP26MTNUPMPKMSBUKNNVC4PJR24UW2NKPLUVEPJBY57KYSQ, seg_pos={0 0}, upload=526138]: Uploading piece... Dec 30 15:47:22.211488991 (600h18m22.082572026s) [noise=true , address=77.90.31.16:4736 , piece_id=R7YJLOWLESSQOR5QI6EDTZPNW764K57E5P5PPTDNCERCSDR4H4MA, seg_pos={0 0}, upload=526138]: Uploading piece... Dec 30 15:47:22.211699963 (600h18m22.082783008s) [noise=true , address=145.79.227.5:4223 , piece_id=7EDBCENJ42MV46PKYISCSSD6WOLJBAWKC54R72LOHJ2ROBMP6BKA, seg_pos={0 0}, upload=526138]: Uploading piece... Dec 30 15:47:22.211889894 (600h18m22.082972929s) [noise=true , address=145.79.227.4:4187 , piece_id=B27CKFMW3YZ3DL6FIG3YJM3V5WAGNGPBSSBJTPG72MGAETXBO6SA, seg_pos={0 0}, upload=526138]: Uploading piece... Dec 30 15:47:22.211966369 (600h18m22.083049404s) [noise=true , address=145.79.227.16:4909 , piece_id=DYW66U7E6R2S5XC5NOG3HEGERYBWGTT4L752I6GP377F2YHYY63Q, seg_pos={0 0}, upload=526138]: Uploading piece... Dec 30 15:47:22.212271901 (600h18m22.083354936s) [noise=true , address=45.91.116.6:4975 , piece_id=IN5UH4JCQNMYNYACZH6QFMXLRTQSQC2NAIH7AMN6EETKVHBINCRQ, seg_pos={0 0}, upload=526138]: Uploading piece... Dec 30 15:47:22.212370479 (600h18m22.083453514s) [noise=true , address=45.91.116.13:4154 , piece_id=BBPI6XXDURZ6AP2UZ3ZXOMMVXFBPZZSUIVP5XQ4MT6C5WWGCBNOA, seg_pos={0 0}, upload=526138]: Uploading piece... Dec 30 15:47:22.212563736 (600h18m22.083646772s) [noise=true , address=145.79.227.8:4330 , piece_id=KCWNDVLQDG5SDBEWB7LGR3FPFP7E2VTFVT5EF2C5T46LCE2Q53IQ, seg_pos={0 0}, upload=526138]: Uploading piece... Dec 30 15:47:22.212850213 (600h18m22.083933248s) [noise=true , address=45.91.116.9:4264 , piece_id=PRK3UQXBFVHR7OPWABTHLDPHP4IO7NE3WBQHUM3CFKGZOWCXTOCQ, seg_pos={0 0}, upload=526138]: Uploading piece... Dec 30 15:47:22.212867124 (600h18m22.08395017s) [noise=true , address=145.79.227.14:4838 , piece_id=G7QUIO4BBBWYTKDM4FSMYK35WJ65NPRFYUEW3RA7IS5OUF2EZSYQ, seg_pos={0 0}, upload=526138]: Uploading piece... Dec 30 15:47:22.213288948 (600h18m22.084371983s) [noise=true , address=77.90.31.10:4518 , piece_id=NWXBD7LZGDNX6SKEUYWH7SKTYCL5FCNVWT6POUMS2XBSSNEN4SVQ, seg_pos={0 0}, upload=526138]: Uploading piece... Dec 30 15:47:22.213393086 (600h18m22.084476121s) [noise=true , address=145.79.227.23:4547 , piece_id=ABNPGF472TCPZPPPS6JP2T5KJOA5LUB6RHYW7DCHJBZYQXIW7QIA, seg_pos={0 0}, upload=526138]: Uploading piece... Dec 30 15:47:22.213868762 (600h18m22.084951797s) [noise=true , address=145.79.227.32:4835 , piece_id=JB6DISB4WJRN6FJEOX4WJOSJ36K3KRJPLFE3HA55YZ4745IQIHKA, seg_pos={0 0}, upload=526138]: Uploading piece... Dec 30 15:47:22.214451822 (600h18m22.085534858s) [noise=true , address=145.79.227.26:4653 , piece_id=D7B3WDZ32R7JV7NHYARUU5HTDIW73FEC5CB2XDFCH5GXINDWEXWA, seg_pos={0 0}, upload=526138]: Uploading piece... Dec 30 15:47:23.917517925 (600h18m23.78860097s) [upload=526138]: Next returned nil segment Dec 30 15:47:23.917533183 (600h18m23.788616219s) [upload=526138]: Waiting for error group managing segments... Dec 30 15:47:24.203785913 (600h18m24.074868948s) [noise=true , address=145.79.227.32:4835 , piece_id=JB6DISB4WJRN6FJEOX4WJOSJ36K3KRJPLFE3HA55YZ4745IQIHKA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil> Dec 30 15:47:24.205416468 (600h18m24.076499513s) [noise=true , address=145.79.227.26:4653 , piece_id=D7B3WDZ32R7JV7NHYARUU5HTDIW73FEC5CB2XDFCH5GXINDWEXWA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil> Dec 30 15:47:24.210592534 (600h18m24.081675579s) [noise=true , address=145.79.227.14:4838 , piece_id=G7QUIO4BBBWYTKDM4FSMYK35WJ65NPRFYUEW3RA7IS5OUF2EZSYQ, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil> Dec 30 15:47:24.216838858 (600h18m24.087921893s) [noise=true , address=145.79.227.4:4187 , piece_id=B27CKFMW3YZ3DL6FIG3YJM3V5WAGNGPBSSBJTPG72MGAETXBO6SA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil> Dec 30 15:47:24.220417232 (600h18m24.091500267s) [noise=true , address=145.79.227.8:4330 , piece_id=KCWNDVLQDG5SDBEWB7LGR3FPFP7E2VTFVT5EF2C5T46LCE2Q53IQ, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil> Dec 30 15:47:24.263722682 (600h18m24.134805717s) [noise=true , address=145.79.227.51:4409 , piece_id=Y3D6WOP26MTNUPMPKMSBUKNNVC4PJR24UW2NKPLUVEPJBY57KYSQ, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil> Dec 30 15:47:24.278838966 (600h18m24.149922011s) [noise=true , address=145.79.227.16:4909 , piece_id=DYW66U7E6R2S5XC5NOG3HEGERYBWGTT4L752I6GP377F2YHYY63Q, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil> Dec 30 15:47:24.286414761 (600h18m24.157497796s) [noise=true , address=145.79.227.5:4223 , piece_id=7EDBCENJ42MV46PKYISCSSD6WOLJBAWKC54R72LOHJ2ROBMP6BKA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil> Dec 30 15:47:24.297480492 (600h18m24.168563527s) [noise=true , address=145.79.227.23:4547 , piece_id=ABNPGF472TCPZPPPS6JP2T5KJOA5LUB6RHYW7DCHJBZYQXIW7QIA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil> Dec 30 15:47:24.365770968 (600h18m24.236854013s) [noise=true , address=77.90.31.16:4736 , piece_id=R7YJLOWLESSQOR5QI6EDTZPNW764K57E5P5PPTDNCERCSDR4H4MA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil> Dec 30 15:47:24.382502307 (600h18m24.253585352s) [noise=true , address=77.90.31.14:4662 , piece_id=FFTHIKWMLVVXVH7GQJH7IRHO2MM2KVM2CIOWAOAAOF3ZD4MSKAGA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil> Dec 30 15:47:24.534833248 (600h18m24.405916283s) [noise=true , address=45.91.116.6:4975 , piece_id=IN5UH4JCQNMYNYACZH6QFMXLRTQSQC2NAIH7AMN6EETKVHBINCRQ, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil> Dec 30 15:47:24.605074860 (600h18m24.476157895s) [noise=true , address=77.90.31.10:4518 , piece_id=NWXBD7LZGDNX6SKEUYWH7SKTYCL5FCNVWT6POUMS2XBSSNEN4SVQ, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil> Dec 30 15:47:24.605094157 (600h18m24.476177202s) [seg_pos={0 0}, upload=526138]: Segment reached optimal threshold of 13 pieces. Dec 30 15:47:24.605387645 (600h18m24.47647068s) [noise=true , address=45.91.116.9:4264 , piece_id=PRK3UQXBFVHR7OPWABTHLDPHP4IO7NE3WBQHUM3CFKGZOWCXTOCQ, seg_pos={0 0}, upload=526138]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:43624->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:43624->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43624->45.91.116.9:4264: use of closed network connection Dec 30 15:47:24.605394879 (600h18m24.476477914s) [noise=true , address=45.91.116.9:4264 , piece_id=PRK3UQXBFVHR7OPWABTHLDPHP4IO7NE3WBQHUM3CFKGZOWCXTOCQ, seg_pos={0 0}, upload=526138]: Long tail cancellation due to optimal threshold Dec 30 15:47:24.606051610 (600h18m24.477134675s) [noise=true , address=77.90.31.11:4556 , piece_id=J27PY5SWPMLPZPCNDBIBV62G7TQPF7W7RVSW74EIKVSFGXZQV5TA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 30 15:47:24.606103147 (600h18m24.477186202s) [noise=true , address=77.90.31.11:4556 , piece_id=J27PY5SWPMLPZPCNDBIBV62G7TQPF7W7RVSW74EIKVSFGXZQV5TA, seg_pos={0 0}, upload=526138]: Long tail cancellation due to optimal threshold Dec 30 15:47:24.606360237 (600h18m24.477443272s) [noise=true , address=45.91.116.13:4154 , piece_id=BBPI6XXDURZ6AP2UZ3ZXOMMVXFBPZZSUIVP5XQ4MT6C5WWGCBNOA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 30 15:47:24.606366910 (600h18m24.477449945s) [noise=true , address=45.91.116.13:4154 , piece_id=BBPI6XXDURZ6AP2UZ3ZXOMMVXFBPZZSUIVP5XQ4MT6C5WWGCBNOA, seg_pos={0 0}, upload=526138]: Long tail cancellation due to optimal threshold Dec 30 15:47:24.606384434 (600h18m24.477467469s) [seg_pos={0 0}, upload=526138]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 15:47:24.606456280 (600h18m24.477539316s) [upload=526138]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 15:47:24.606462161 (600h18m24.477545197s) [upload=526138]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 15:47:24.892704121 (600h18m24.763787166s) [upload=526138]: Done uploading segments... Dec 30 15:47:24.892747804 (600h18m24.763830849s) [upload=526138]: Upload finished. err: <nil> Duration: 5.833360363s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 44 out of 60 is failed (threshold=6) [X.XXXXX..XXXX.XX.X.X.X.XXXXXX.XXXX...XX.X.XXXXXXXXX.XXXXXXX.] duration: 5.001339009s trace: https://jaeger.dev.storj.io/trace/2d8479babd245c3e FAIL (2025-12-30 15:51:06 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001362362s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........X............................X..XX..X.............] duration: 54.779356171s trace: https://jaeger.dev.storj.io/trace/3b27a789ef618091 FAIL (2025-12-30 15:51:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ise=true , address=145.79.227.5:4252 , piece_id=JRKV35AO742VMYZPN3WKPNRZKO2ZFUU53RS4MXF22HO5UC6BKCOQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.081840064 (600h10m41.952923099s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.082191573 (600h10m41.953274608s) [noise=true , address=145.79.227.16:4942 , piece_id=G5I53KG36VHVW5LICAHOHRXTQVQ67ZFGJQNXRJMYNOIT7WB7XTEQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.082429897 (600h10m41.953512932s) [noise=true , address=145.79.227.29:4968 , piece_id=PMTNKRHI6YWMJPZ2R7RPIF7Q6CFLFKBH5OTHWCX4RNGIAFURVMOQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.082680584 (600h10m41.953763619s) [noise=true , address=145.79.227.33:4892 , piece_id=RSEJC3XHFACGLV6YBHYM7QBCU7MMYST53FJO4ITLA2C3IHBZCRTQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.082917976 (600h10m41.954001011s) [noise=true , address=77.90.31.12:4596 , piece_id=TQVNHTTAWVUCUIHSHWWH27MFJNOE2MWLJCRMTPUCREML7DT3RUYQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.083149276 (600h10m41.954232311s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.083393431 (600h10m41.954476466s) [noise=true , address=145.79.227.35:4508 , piece_id=KJJFNTPYUUHLFFCGHD3KDTN3DK6C2R5YRLWJDMF6C4UGSMFYVFBA, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.083842486 (600h10m41.954925521s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.086214362 (600h10m41.957297397s) [noise=true , address=145.79.227.49:4367 , piece_id=QUI5FSOTJVIHT67I3F2YDB5IPKSYEYJTDRKA44UM355S5FSNNLUA, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.086612852 (600h10m41.957695887s) [noise=true , address=77.90.31.13:4652 , piece_id=5VR2F6XUEYGX5KULBIL6V3YGH7Q3T5UYFJTS5RGWN3B7F7TVMHEQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.087061346 (600h10m41.958144381s) [noise=true , address=77.90.31.11:4588 , piece_id=WY5XS4EPFHDLCT3XEQQKBO2M5GLVTBNDZOKPTFPVZV6EALPMNSHA, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.087452310 (600h10m41.958535345s) [noise=true , address=145.79.227.30:4768 , piece_id=2KZKXIYGSDES5Q5W2PVAACJYU6ANKQNDP2VQEBXM7QKXOT6QLSBA, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.087903780 (600h10m41.958986825s) [noise=true , address=64.49.13.27:4957 , piece_id=ETFKCNNSFAVJWNB3J3U4LPRDGBILNNA5CK7FR7X5UHONPZHXFBLQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.088521366 (600h10m41.959604411s) [noise=true , address=193.149.164.143:4698 , piece_id=F6B3UPOKCSD7ZXQ4SYKOAALEBA4FSTQ2ELGMGCU7TMWVQZDZB2SQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.088618600 (600h10m41.959701645s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:43.511091675 (600h10m43.38217472s) [upload=526030]: Next returned nil segment Dec 30 15:39:43.511109379 (600h10m43.382192424s) [upload=526030]: Waiting for error group managing segments... Dec 30 15:39:44.017333248 (600h10m43.888416283s) [noise=true , address=145.79.227.5:4252 , piece_id=JRKV35AO742VMYZPN3WKPNRZKO2ZFUU53RS4MXF22HO5UC6BKCOQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.069300150 (600h10m43.940383186s) [noise=true , address=64.49.13.27:4957 , piece_id=ETFKCNNSFAVJWNB3J3U4LPRDGBILNNA5CK7FR7X5UHONPZHXFBLQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.126868228 (600h10m43.997951273s) [noise=true , address=77.90.31.17:4800 , piece_id=PYZVLAZZKWP6UOL6HJVVWSEJOXCJWK5IAMWJPRODOKCLRZE37BAQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.146896235 (600h10m44.01797927s) [noise=true , address=145.79.227.29:4968 , piece_id=PMTNKRHI6YWMJPZ2R7RPIF7Q6CFLFKBH5OTHWCX4RNGIAFURVMOQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.166221683 (600h10m44.037304718s) [noise=true , address=77.90.31.12:4596 , piece_id=TQVNHTTAWVUCUIHSHWWH27MFJNOE2MWLJCRMTPUCREML7DT3RUYQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.234530763 (600h10m44.105613808s) [noise=true , address=77.90.31.13:4652 , piece_id=5VR2F6XUEYGX5KULBIL6V3YGH7Q3T5UYFJTS5RGWN3B7F7TVMHEQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.489468910 (600h10m44.360551955s) [noise=true , address=145.79.227.16:4942 , piece_id=G5I53KG36VHVW5LICAHOHRXTQVQ67ZFGJQNXRJMYNOIT7WB7XTEQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.802041331 (600h10m44.673124366s) [noise=true , address=193.149.164.143:4698 , piece_id=F6B3UPOKCSD7ZXQ4SYKOAALEBA4FSTQ2ELGMGCU7TMWVQZDZB2SQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.822093824 (600h10m44.693176859s) [noise=true , address=145.79.227.49:4367 , piece_id=QUI5FSOTJVIHT67I3F2YDB5IPKSYEYJTDRKA44UM355S5FSNNLUA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.867276618 (600h10m44.738359653s) [noise=true , address=145.79.227.30:4768 , piece_id=2KZKXIYGSDES5Q5W2PVAACJYU6ANKQNDP2VQEBXM7QKXOT6QLSBA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.881025168 (600h10m44.752108203s) [noise=true , address=145.79.227.44:4199 , piece_id=AJKADWXHHMCPE6V52UMJJPCHOBBL7QDU7DF337LB6Q3YDHEYUUQQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.899437537 (600h10m44.770520572s) [noise=true , address=77.90.31.11:4588 , piece_id=WY5XS4EPFHDLCT3XEQQKBO2M5GLVTBNDZOKPTFPVZV6EALPMNSHA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.941939747 (600h10m44.813022782s) [noise=true , address=145.79.227.33:4892 , piece_id=RSEJC3XHFACGLV6YBHYM7QBCU7MMYST53FJO4ITLA2C3IHBZCRTQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.980395983 (600h10m44.851479029s) [noise=true , address=145.79.227.35:4508 , piece_id=KJJFNTPYUUHLFFCGHD3KDTN3DK6C2R5YRLWJDMF6C4UGSMFYVFBA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.980415871 (600h10m44.851498916s) [seg_pos={0 0}, upload=526030]: Segment reached optimal threshold of 14 pieces. Dec 30 15:39:44.980578861 (600h10m44.851661896s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection Dec 30 15:39:44.980584171 (600h10m44.851667206s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold Dec 30 15:39:44.980730901 (600h10m44.851813936s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 30 15:39:44.980747002 (600h10m44.851830037s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold Dec 30 15:39:44.980793770 (600h10m44.851876806s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127NKToES711diH4f8TFJbwERq3NhhFEFrQyi7Mka3VVLsamuzp): protocol: expected piece hash; context canceled; EOF Dec 30 15:39:44.980799651 (600h10m44.851882687s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold Dec 30 15:39:44.980930771 (600h10m44.852013806s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nnXd911ApLeiKnc4LgDEUNnnkxtnLEdwzP48BhuhyEykacV6W): protocol: expected piece hash; context canceled; EOF Dec 30 15:39:44.980937775 (600h10m44.85202081s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold Dec 30 15:39:44.980956481 (600h10m44.852039526s) [seg_pos={0 0}, upload=526030]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 15:39:44.981039398 (600h10m44.852122433s) [upload=526030]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 15:39:44.981056070 (600h10m44.852139105s) [upload=526030]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 15:39:45.290276710 (600h10m45.161359765s) [upload=526030]: Done uploading segments... Dec 30 15:39:45.290299182 (600h10m45.161382227s) [upload=526030]: Upload finished. err: <nil> Duration: 54.779374767s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X...X.......X.X..........XX.......XX.XX..X....X.......X.....] duration: 5.987241699s trace: https://jaeger.dev.storj.io/trace/161ad17247435e8e FAIL (2025-12-30 15:50:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: FFTHIKWMLVVXVH7GQJH7IRHO2MM2KVM2CIOWAOAAOF3ZD4MSKAGA) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: R7YJLOWLESSQOR5QI6EDTZPNW764K57E5P5PPTDNCERCSDR4H4MA) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: IN5UH4JCQNMYNYACZH6QFMXLRTQSQC2NAIH7AMN6EETKVHBINCRQ) context deadline exceeded ======LOGS====== 30 15:47:21.962125072 (600h18m21.833208107s) [upload=526138]: Starting upload Dec 30 15:47:21.962182793 (600h18m21.833265828s) [upload=526138]: Done starting upload Dec 30 15:47:21.962460822 (600h18m21.833543857s) [upload=526138]: Uploading segments... Dec 30 15:47:21.963187436 (600h18m21.834270481s) [upload=526138]: Got next segment. Inline: false Dec 30 15:47:21.963245817 (600h18m21.834328882s) [upload=526138]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 15:47:21.963260635 (600h18m21.83434367s) [upload=526138]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 15:47:22.210595890 (600h18m22.081678935s) [seg_pos={0 0}, upload=526138]: Begin upload segment... Dec 30 15:47:22.210645103 (600h18m22.081728148s) [seg_pos={0 0}, upload=526138]: Done begin upload segment. Dec 30 15:47:22.210709887 (600h18m22.081792932s) [noise=true , address=77.90.31.11:4556 , piece_id=J27PY5SWPMLPZPCNDBIBV62G7TQPF7W7RVSW74EIKVSFGXZQV5TA, seg_pos={0 0}, upload=526138]: Uploading piece... Dec 30 15:47:22.210732190 (600h18m22.081815225s) [noise=true , address=77.90.31.14:4662 , piece_id=FFTHIKWMLVVXVH7GQJH7IRHO2MM2KVM2CIOWAOAAOF3ZD4MSKAGA, seg_pos={0 0}, upload=526138]: Uploading piece... Dec 30 15:47:22.211406824 (600h18m22.082489859s) [noise=true , address=145.79.227.51:4409 , piece_id=Y3D6WOP26MTNUPMPKMSBUKNNVC4PJR24UW2NKPLUVEPJBY57KYSQ, seg_pos={0 0}, upload=526138]: Uploading piece... Dec 30 15:47:22.211488991 (600h18m22.082572026s) [noise=true , address=77.90.31.16:4736 , piece_id=R7YJLOWLESSQOR5QI6EDTZPNW764K57E5P5PPTDNCERCSDR4H4MA, seg_pos={0 0}, upload=526138]: Uploading piece... Dec 30 15:47:22.211699963 (600h18m22.082783008s) [noise=true , address=145.79.227.5:4223 , piece_id=7EDBCENJ42MV46PKYISCSSD6WOLJBAWKC54R72LOHJ2ROBMP6BKA, seg_pos={0 0}, upload=526138]: Uploading piece... Dec 30 15:47:22.211889894 (600h18m22.082972929s) [noise=true , address=145.79.227.4:4187 , piece_id=B27CKFMW3YZ3DL6FIG3YJM3V5WAGNGPBSSBJTPG72MGAETXBO6SA, seg_pos={0 0}, upload=526138]: Uploading piece... Dec 30 15:47:22.211966369 (600h18m22.083049404s) [noise=true , address=145.79.227.16:4909 , piece_id=DYW66U7E6R2S5XC5NOG3HEGERYBWGTT4L752I6GP377F2YHYY63Q, seg_pos={0 0}, upload=526138]: Uploading piece... Dec 30 15:47:22.212271901 (600h18m22.083354936s) [noise=true , address=45.91.116.6:4975 , piece_id=IN5UH4JCQNMYNYACZH6QFMXLRTQSQC2NAIH7AMN6EETKVHBINCRQ, seg_pos={0 0}, upload=526138]: Uploading piece... Dec 30 15:47:22.212370479 (600h18m22.083453514s) [noise=true , address=45.91.116.13:4154 , piece_id=BBPI6XXDURZ6AP2UZ3ZXOMMVXFBPZZSUIVP5XQ4MT6C5WWGCBNOA, seg_pos={0 0}, upload=526138]: Uploading piece... Dec 30 15:47:22.212563736 (600h18m22.083646772s) [noise=true , address=145.79.227.8:4330 , piece_id=KCWNDVLQDG5SDBEWB7LGR3FPFP7E2VTFVT5EF2C5T46LCE2Q53IQ, seg_pos={0 0}, upload=526138]: Uploading piece... Dec 30 15:47:22.212850213 (600h18m22.083933248s) [noise=true , address=45.91.116.9:4264 , piece_id=PRK3UQXBFVHR7OPWABTHLDPHP4IO7NE3WBQHUM3CFKGZOWCXTOCQ, seg_pos={0 0}, upload=526138]: Uploading piece... Dec 30 15:47:22.212867124 (600h18m22.08395017s) [noise=true , address=145.79.227.14:4838 , piece_id=G7QUIO4BBBWYTKDM4FSMYK35WJ65NPRFYUEW3RA7IS5OUF2EZSYQ, seg_pos={0 0}, upload=526138]: Uploading piece... Dec 30 15:47:22.213288948 (600h18m22.084371983s) [noise=true , address=77.90.31.10:4518 , piece_id=NWXBD7LZGDNX6SKEUYWH7SKTYCL5FCNVWT6POUMS2XBSSNEN4SVQ, seg_pos={0 0}, upload=526138]: Uploading piece... Dec 30 15:47:22.213393086 (600h18m22.084476121s) [noise=true , address=145.79.227.23:4547 , piece_id=ABNPGF472TCPZPPPS6JP2T5KJOA5LUB6RHYW7DCHJBZYQXIW7QIA, seg_pos={0 0}, upload=526138]: Uploading piece... Dec 30 15:47:22.213868762 (600h18m22.084951797s) [noise=true , address=145.79.227.32:4835 , piece_id=JB6DISB4WJRN6FJEOX4WJOSJ36K3KRJPLFE3HA55YZ4745IQIHKA, seg_pos={0 0}, upload=526138]: Uploading piece... Dec 30 15:47:22.214451822 (600h18m22.085534858s) [noise=true , address=145.79.227.26:4653 , piece_id=D7B3WDZ32R7JV7NHYARUU5HTDIW73FEC5CB2XDFCH5GXINDWEXWA, seg_pos={0 0}, upload=526138]: Uploading piece... Dec 30 15:47:23.917517925 (600h18m23.78860097s) [upload=526138]: Next returned nil segment Dec 30 15:47:23.917533183 (600h18m23.788616219s) [upload=526138]: Waiting for error group managing segments... Dec 30 15:47:24.203785913 (600h18m24.074868948s) [noise=true , address=145.79.227.32:4835 , piece_id=JB6DISB4WJRN6FJEOX4WJOSJ36K3KRJPLFE3HA55YZ4745IQIHKA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil> Dec 30 15:47:24.205416468 (600h18m24.076499513s) [noise=true , address=145.79.227.26:4653 , piece_id=D7B3WDZ32R7JV7NHYARUU5HTDIW73FEC5CB2XDFCH5GXINDWEXWA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil> Dec 30 15:47:24.210592534 (600h18m24.081675579s) [noise=true , address=145.79.227.14:4838 , piece_id=G7QUIO4BBBWYTKDM4FSMYK35WJ65NPRFYUEW3RA7IS5OUF2EZSYQ, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil> Dec 30 15:47:24.216838858 (600h18m24.087921893s) [noise=true , address=145.79.227.4:4187 , piece_id=B27CKFMW3YZ3DL6FIG3YJM3V5WAGNGPBSSBJTPG72MGAETXBO6SA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil> Dec 30 15:47:24.220417232 (600h18m24.091500267s) [noise=true , address=145.79.227.8:4330 , piece_id=KCWNDVLQDG5SDBEWB7LGR3FPFP7E2VTFVT5EF2C5T46LCE2Q53IQ, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil> Dec 30 15:47:24.263722682 (600h18m24.134805717s) [noise=true , address=145.79.227.51:4409 , piece_id=Y3D6WOP26MTNUPMPKMSBUKNNVC4PJR24UW2NKPLUVEPJBY57KYSQ, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil> Dec 30 15:47:24.278838966 (600h18m24.149922011s) [noise=true , address=145.79.227.16:4909 , piece_id=DYW66U7E6R2S5XC5NOG3HEGERYBWGTT4L752I6GP377F2YHYY63Q, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil> Dec 30 15:47:24.286414761 (600h18m24.157497796s) [noise=true , address=145.79.227.5:4223 , piece_id=7EDBCENJ42MV46PKYISCSSD6WOLJBAWKC54R72LOHJ2ROBMP6BKA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil> Dec 30 15:47:24.297480492 (600h18m24.168563527s) [noise=true , address=145.79.227.23:4547 , piece_id=ABNPGF472TCPZPPPS6JP2T5KJOA5LUB6RHYW7DCHJBZYQXIW7QIA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil> Dec 30 15:47:24.365770968 (600h18m24.236854013s) [noise=true , address=77.90.31.16:4736 , piece_id=R7YJLOWLESSQOR5QI6EDTZPNW764K57E5P5PPTDNCERCSDR4H4MA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil> Dec 30 15:47:24.382502307 (600h18m24.253585352s) [noise=true , address=77.90.31.14:4662 , piece_id=FFTHIKWMLVVXVH7GQJH7IRHO2MM2KVM2CIOWAOAAOF3ZD4MSKAGA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil> Dec 30 15:47:24.534833248 (600h18m24.405916283s) [noise=true , address=45.91.116.6:4975 , piece_id=IN5UH4JCQNMYNYACZH6QFMXLRTQSQC2NAIH7AMN6EETKVHBINCRQ, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil> Dec 30 15:47:24.605074860 (600h18m24.476157895s) [noise=true , address=77.90.31.10:4518 , piece_id=NWXBD7LZGDNX6SKEUYWH7SKTYCL5FCNVWT6POUMS2XBSSNEN4SVQ, seg_pos={0 0}, upload=526138]: Done uploading piece. err: <nil> Dec 30 15:47:24.605094157 (600h18m24.476177202s) [seg_pos={0 0}, upload=526138]: Segment reached optimal threshold of 13 pieces. Dec 30 15:47:24.605387645 (600h18m24.47647068s) [noise=true , address=45.91.116.9:4264 , piece_id=PRK3UQXBFVHR7OPWABTHLDPHP4IO7NE3WBQHUM3CFKGZOWCXTOCQ, seg_pos={0 0}, upload=526138]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:128f6i7ALVXF3Vr8G8zsZXrNBFiT3cK8UAiic5XbFS1gRjZK86m): protocol: write tcp 188.166.225.170:43624->45.91.116.9:4264: use of closed network connection; write tcp 188.166.225.170:43624->45.91.116.9:4264: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43624->45.91.116.9:4264: use of closed network connection Dec 30 15:47:24.605394879 (600h18m24.476477914s) [noise=true , address=45.91.116.9:4264 , piece_id=PRK3UQXBFVHR7OPWABTHLDPHP4IO7NE3WBQHUM3CFKGZOWCXTOCQ, seg_pos={0 0}, upload=526138]: Long tail cancellation due to optimal threshold Dec 30 15:47:24.606051610 (600h18m24.477134675s) [noise=true , address=77.90.31.11:4556 , piece_id=J27PY5SWPMLPZPCNDBIBV62G7TQPF7W7RVSW74EIKVSFGXZQV5TA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: expected piece hash; context canceled; EOF Dec 30 15:47:24.606103147 (600h18m24.477186202s) [noise=true , address=77.90.31.11:4556 , piece_id=J27PY5SWPMLPZPCNDBIBV62G7TQPF7W7RVSW74EIKVSFGXZQV5TA, seg_pos={0 0}, upload=526138]: Long tail cancellation due to optimal threshold Dec 30 15:47:24.606360237 (600h18m24.477443272s) [noise=true , address=45.91.116.13:4154 , piece_id=BBPI6XXDURZ6AP2UZ3ZXOMMVXFBPZZSUIVP5XQ4MT6C5WWGCBNOA, seg_pos={0 0}, upload=526138]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 30 15:47:24.606366910 (600h18m24.477449945s) [noise=true , address=45.91.116.13:4154 , piece_id=BBPI6XXDURZ6AP2UZ3ZXOMMVXFBPZZSUIVP5XQ4MT6C5WWGCBNOA, seg_pos={0 0}, upload=526138]: Long tail cancellation due to optimal threshold Dec 30 15:47:24.606384434 (600h18m24.477467469s) [seg_pos={0 0}, upload=526138]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 15:47:24.606456280 (600h18m24.477539316s) [upload=526138]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 15:47:24.606462161 (600h18m24.477545197s) [upload=526138]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 15:47:24.892704121 (600h18m24.763787166s) [upload=526138]: Done uploading segments... Dec 30 15:47:24.892747804 (600h18m24.763830849s) [upload=526138]: Upload finished. err: <nil> Duration: 5.987267738s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 45 out of 60 is failed (threshold=6) [XX.XXXXX..XXXX.XX.X.X.X.XXXXXX.XXXX...XX.X.XXXXXXXXX.XXXXXXX] duration: 5.001511006s trace: https://jaeger.dev.storj.io/trace/2d8479babd245c3e FAIL (2025-12-30 15:50:49 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001528738s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 13.057770476s trace: https://jaeger.dev.storj.io/trace/547648a2cfce50e7 FAIL (2025-12-30 15:50:26 UTC) orbiter warning: context deadline exceeded Duration: 13.057796044s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........X............................X..XX..X............] duration: 37.985251517s trace: https://jaeger.dev.storj.io/trace/3b27a789ef618091 FAIL (2025-12-30 15:50:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ise=true , address=145.79.227.5:4252 , piece_id=JRKV35AO742VMYZPN3WKPNRZKO2ZFUU53RS4MXF22HO5UC6BKCOQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.081840064 (600h10m41.952923099s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.082191573 (600h10m41.953274608s) [noise=true , address=145.79.227.16:4942 , piece_id=G5I53KG36VHVW5LICAHOHRXTQVQ67ZFGJQNXRJMYNOIT7WB7XTEQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.082429897 (600h10m41.953512932s) [noise=true , address=145.79.227.29:4968 , piece_id=PMTNKRHI6YWMJPZ2R7RPIF7Q6CFLFKBH5OTHWCX4RNGIAFURVMOQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.082680584 (600h10m41.953763619s) [noise=true , address=145.79.227.33:4892 , piece_id=RSEJC3XHFACGLV6YBHYM7QBCU7MMYST53FJO4ITLA2C3IHBZCRTQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.082917976 (600h10m41.954001011s) [noise=true , address=77.90.31.12:4596 , piece_id=TQVNHTTAWVUCUIHSHWWH27MFJNOE2MWLJCRMTPUCREML7DT3RUYQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.083149276 (600h10m41.954232311s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.083393431 (600h10m41.954476466s) [noise=true , address=145.79.227.35:4508 , piece_id=KJJFNTPYUUHLFFCGHD3KDTN3DK6C2R5YRLWJDMF6C4UGSMFYVFBA, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.083842486 (600h10m41.954925521s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.086214362 (600h10m41.957297397s) [noise=true , address=145.79.227.49:4367 , piece_id=QUI5FSOTJVIHT67I3F2YDB5IPKSYEYJTDRKA44UM355S5FSNNLUA, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.086612852 (600h10m41.957695887s) [noise=true , address=77.90.31.13:4652 , piece_id=5VR2F6XUEYGX5KULBIL6V3YGH7Q3T5UYFJTS5RGWN3B7F7TVMHEQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.087061346 (600h10m41.958144381s) [noise=true , address=77.90.31.11:4588 , piece_id=WY5XS4EPFHDLCT3XEQQKBO2M5GLVTBNDZOKPTFPVZV6EALPMNSHA, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.087452310 (600h10m41.958535345s) [noise=true , address=145.79.227.30:4768 , piece_id=2KZKXIYGSDES5Q5W2PVAACJYU6ANKQNDP2VQEBXM7QKXOT6QLSBA, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.087903780 (600h10m41.958986825s) [noise=true , address=64.49.13.27:4957 , piece_id=ETFKCNNSFAVJWNB3J3U4LPRDGBILNNA5CK7FR7X5UHONPZHXFBLQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.088521366 (600h10m41.959604411s) [noise=true , address=193.149.164.143:4698 , piece_id=F6B3UPOKCSD7ZXQ4SYKOAALEBA4FSTQ2ELGMGCU7TMWVQZDZB2SQ, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:42.088618600 (600h10m41.959701645s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Uploading piece... Dec 30 15:39:43.511091675 (600h10m43.38217472s) [upload=526030]: Next returned nil segment Dec 30 15:39:43.511109379 (600h10m43.382192424s) [upload=526030]: Waiting for error group managing segments... Dec 30 15:39:44.017333248 (600h10m43.888416283s) [noise=true , address=145.79.227.5:4252 , piece_id=JRKV35AO742VMYZPN3WKPNRZKO2ZFUU53RS4MXF22HO5UC6BKCOQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.069300150 (600h10m43.940383186s) [noise=true , address=64.49.13.27:4957 , piece_id=ETFKCNNSFAVJWNB3J3U4LPRDGBILNNA5CK7FR7X5UHONPZHXFBLQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.126868228 (600h10m43.997951273s) [noise=true , address=77.90.31.17:4800 , piece_id=PYZVLAZZKWP6UOL6HJVVWSEJOXCJWK5IAMWJPRODOKCLRZE37BAQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.146896235 (600h10m44.01797927s) [noise=true , address=145.79.227.29:4968 , piece_id=PMTNKRHI6YWMJPZ2R7RPIF7Q6CFLFKBH5OTHWCX4RNGIAFURVMOQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.166221683 (600h10m44.037304718s) [noise=true , address=77.90.31.12:4596 , piece_id=TQVNHTTAWVUCUIHSHWWH27MFJNOE2MWLJCRMTPUCREML7DT3RUYQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.234530763 (600h10m44.105613808s) [noise=true , address=77.90.31.13:4652 , piece_id=5VR2F6XUEYGX5KULBIL6V3YGH7Q3T5UYFJTS5RGWN3B7F7TVMHEQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.489468910 (600h10m44.360551955s) [noise=true , address=145.79.227.16:4942 , piece_id=G5I53KG36VHVW5LICAHOHRXTQVQ67ZFGJQNXRJMYNOIT7WB7XTEQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.802041331 (600h10m44.673124366s) [noise=true , address=193.149.164.143:4698 , piece_id=F6B3UPOKCSD7ZXQ4SYKOAALEBA4FSTQ2ELGMGCU7TMWVQZDZB2SQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.822093824 (600h10m44.693176859s) [noise=true , address=145.79.227.49:4367 , piece_id=QUI5FSOTJVIHT67I3F2YDB5IPKSYEYJTDRKA44UM355S5FSNNLUA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.867276618 (600h10m44.738359653s) [noise=true , address=145.79.227.30:4768 , piece_id=2KZKXIYGSDES5Q5W2PVAACJYU6ANKQNDP2VQEBXM7QKXOT6QLSBA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.881025168 (600h10m44.752108203s) [noise=true , address=145.79.227.44:4199 , piece_id=AJKADWXHHMCPE6V52UMJJPCHOBBL7QDU7DF337LB6Q3YDHEYUUQQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.899437537 (600h10m44.770520572s) [noise=true , address=77.90.31.11:4588 , piece_id=WY5XS4EPFHDLCT3XEQQKBO2M5GLVTBNDZOKPTFPVZV6EALPMNSHA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.941939747 (600h10m44.813022782s) [noise=true , address=145.79.227.33:4892 , piece_id=RSEJC3XHFACGLV6YBHYM7QBCU7MMYST53FJO4ITLA2C3IHBZCRTQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.980395983 (600h10m44.851479029s) [noise=true , address=145.79.227.35:4508 , piece_id=KJJFNTPYUUHLFFCGHD3KDTN3DK6C2R5YRLWJDMF6C4UGSMFYVFBA, seg_pos={0 0}, upload=526030]: Done uploading piece. err: <nil> Dec 30 15:39:44.980415871 (600h10m44.851498916s) [seg_pos={0 0}, upload=526030]: Segment reached optimal threshold of 14 pieces. Dec 30 15:39:44.980578861 (600h10m44.851661896s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1C8U9EuxPw2xxuH8rkdoSv4nGaoBB7syRzrZYchYhoQYHxjQSD): protocol: write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection; write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49762->64.49.13.169:4707: use of closed network connection Dec 30 15:39:44.980584171 (600h10m44.851667206s) [noise=true , address=64.49.13.169:4707 , piece_id=5NHKQJBQWSIMTFX47MKGNGAGMHKS5ENFOL3ROY45FLLXS7NSVJLQ, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold Dec 30 15:39:44.980730901 (600h10m44.851813936s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Nw78dd9WPnmCwv5bFQR3VpV39776Pye2QVGSNMTs3zq2BtaGJ): protocol: expected piece hash; context canceled; EOF Dec 30 15:39:44.980747002 (600h10m44.851830037s) [noise=true , address=145.79.227.34:4918 , piece_id=6BBY7R4FCE7CW4EJG4AMINHO5YDA3KUJH4TAC7QNHXKBHL7C3M6A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold Dec 30 15:39:44.980793770 (600h10m44.851876806s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:127NKToES711diH4f8TFJbwERq3NhhFEFrQyi7Mka3VVLsamuzp): protocol: expected piece hash; context canceled; EOF Dec 30 15:39:44.980799651 (600h10m44.851882687s) [noise=true , address=145.79.227.40:4065 , piece_id=6FSURDUGYJU4CE6LEWXTNFEIGP7X7GFXPUTO75PAQWNQKV3T2H2A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold Dec 30 15:39:44.980930771 (600h10m44.852013806s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12nnXd911ApLeiKnc4LgDEUNnnkxtnLEdwzP48BhuhyEykacV6W): protocol: expected piece hash; context canceled; EOF Dec 30 15:39:44.980937775 (600h10m44.85202081s) [noise=true , address=77.90.31.12:4612 , piece_id=W5IMLESMOEXZGSYXTUFENV5JHPDZKN4J2BYOZ3EEI5P4PDDQUM7A, seg_pos={0 0}, upload=526030]: Long tail cancellation due to optimal threshold Dec 30 15:39:44.980956481 (600h10m44.852039526s) [seg_pos={0 0}, upload=526030]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 15:39:44.981039398 (600h10m44.852122433s) [upload=526030]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 15:39:44.981056070 (600h10m44.852139105s) [upload=526030]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 15:39:45.290276710 (600h10m45.161359765s) [upload=526030]: Done uploading segments... Dec 30 15:39:45.290299182 (600h10m45.161382227s) [upload=526030]: Upload finished. err: <nil> Duration: 37.985281785s core uplink us-select-1 us1-placement-20

1
...
220
221
222
223
224
...
7138