Orbiter History

Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 357600 reports (Page 225 of 7152)

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...X..X............................XX.......................] duration: 6.019931558s trace: https://jaeger.dev.storj.io/trace/4173ea2fcd946b89 FAIL (2025-12-30 17:52:44 UTC) orbiter warning: context deadline exceeded Duration: 6.020081511s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 29.513469342s trace: https://jaeger.dev.storj.io/trace/5ea8d7d4f61e13a2 FAIL (2025-12-30 17:52:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1X5A7bLqMEQuaMbf9xZ7PDUD4161sz2QqdM9r59y3bsyPBJhaK, Piece ID: HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ) context deadline exceeded; piecestore: (Node ID: 12eLKUg2vTnZMrLaraSPEoGXe3jDVBJYyURdcn4f2gCN5pvvCQ8, Piece ID: KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ) context deadline exceeded; piecestore: (Node ID: 12W4n6uf6azQk1QgeR8Xw7UkpnZm19zdyFKwojCsqZt2GFHrpce, Piece ID: DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ) context deadline exceeded ======LOGS====== 17.114181822 (601h42m16.985264867s) [noise=true , address=145.79.227.44:4208 , piece_id=3OXDCHZSOIFS5VQCI4SXSAQFICDCYHRE5NGFBERR33KOTGSZWLDQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.114641387 (601h42m16.985724432s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.115652532 (601h42m16.986735578s) [noise=true , address=77.90.31.17:4790 , piece_id=5RSRESKGDA2YF4NBRF4ZFABBKWJK4NO6VLEXW2Z6EY3OPXH2WPNA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.115937134 (601h42m16.987020179s) [noise=true , address=64.49.13.155:4923 , piece_id=MUJD7XLM2YRLRCHYVSDPBE6VBYLHCOA7HJQP22ZSET4N574XQMJA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.116814986 (601h42m16.987898031s) [noise=true , address=45.91.116.11:4139 , piece_id=72DPEWZ7ZPZZ53UGYV2VCUKLCK7HCM22CDBRHWVLJGDGO5B5JADQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.117501182 (601h42m16.988584228s) [noise=true , address=77.90.31.14:4665 , piece_id=DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.118263334 (601h42m16.989346379s) [noise=true , address=145.79.227.40:4084 , piece_id=ZADZDRKBIILLAI4IT335CYETOSFOFUUPQIH6HZJCTHKRTZSPDJUQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.118555850 (601h42m16.989638886s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.118765930 (601h42m16.989849036s) [noise=true , address=77.90.31.21:4905 , piece_id=HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.119029533 (601h42m16.990112568s) [noise=true , address=77.90.31.10:4533 , piece_id=QFBBYNQHQL2G7PEREK43FZBRUMKMVNBC2XKTE2NWTVTKN2452LPA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.119417672 (601h42m16.990500707s) [noise=true , address=45.91.116.12:4258 , piece_id=KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.119441588 (601h42m16.990524643s) [noise=true , address=145.79.227.40:4053 , piece_id=JCSIUMWKEAQNQXTNSJTN6VSB3V2SIDRQK67QMK46B4GOUWEMQE2A, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.119763470 (601h42m16.990846505s) [noise=true , address=77.90.31.18:4821 , piece_id=MXZFKXM3JV2HQURDLIU7BKWLAKDZ27O4ER3SGC7QBZV5I3J2EHKA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.120028685 (601h42m16.99111172s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.120086986 (601h42m16.991170031s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.120586507 (601h42m16.991669562s) [noise=true , address=45.91.116.13:4168 , piece_id=AKUE5H3A2UECQOZY3GV3MWZKIHSNKCDOG32DKCNHIRTJROJ74M3A, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.121258768 (601h42m16.992341813s) [noise=true , address=145.79.227.9:4379 , piece_id=QTFXFSGGKYBQSPWJYBBJMQUL3D5EEO3IX4C53ECD7FXMUOGUBC7Q, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:18.568739165 (601h42m18.43982221s) [upload=527370]: Next returned nil segment Dec 30 17:11:18.568751328 (601h42m18.439834363s) [upload=527370]: Waiting for error group managing segments... Dec 30 17:11:18.978424178 (601h42m18.849507213s) [noise=true , address=64.49.13.155:4923 , piece_id=MUJD7XLM2YRLRCHYVSDPBE6VBYLHCOA7HJQP22ZSET4N574XQMJA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:18.986258125 (601h42m18.85734116s) [noise=true , address=145.79.227.9:4379 , piece_id=QTFXFSGGKYBQSPWJYBBJMQUL3D5EEO3IX4C53ECD7FXMUOGUBC7Q, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.274519421 (601h42m19.145602456s) [noise=true , address=77.90.31.14:4665 , piece_id=DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.315775438 (601h42m19.186858473s) [noise=true , address=45.91.116.13:4168 , piece_id=AKUE5H3A2UECQOZY3GV3MWZKIHSNKCDOG32DKCNHIRTJROJ74M3A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.335765371 (601h42m19.206848406s) [noise=true , address=77.90.31.21:4905 , piece_id=HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.354812129 (601h42m19.225895164s) [noise=true , address=77.90.31.10:4533 , piece_id=QFBBYNQHQL2G7PEREK43FZBRUMKMVNBC2XKTE2NWTVTKN2452LPA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.392024917 (601h42m19.263107952s) [noise=true , address=45.91.116.11:4139 , piece_id=72DPEWZ7ZPZZ53UGYV2VCUKLCK7HCM22CDBRHWVLJGDGO5B5JADQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.396084327 (601h42m19.267167372s) [noise=true , address=77.90.31.18:4821 , piece_id=MXZFKXM3JV2HQURDLIU7BKWLAKDZ27O4ER3SGC7QBZV5I3J2EHKA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.862099099 (601h42m19.733182134s) [noise=true , address=77.90.31.17:4790 , piece_id=5RSRESKGDA2YF4NBRF4ZFABBKWJK4NO6VLEXW2Z6EY3OPXH2WPNA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.905412913 (601h42m19.776495948s) [noise=true , address=145.79.227.44:4208 , piece_id=3OXDCHZSOIFS5VQCI4SXSAQFICDCYHRE5NGFBERR33KOTGSZWLDQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.910309688 (601h42m19.781392743s) [noise=true , address=145.79.227.30:4794 , piece_id=EJH6Q3T2BMPVLG6F7CW7N4GYOA3EZFPADL6VPT7WKKG65B2WZVXQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.934233361 (601h42m19.805316397s) [noise=true , address=145.79.227.40:4084 , piece_id=ZADZDRKBIILLAI4IT335CYETOSFOFUUPQIH6HZJCTHKRTZSPDJUQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:20.074011527 (601h42m19.945094572s) [noise=true , address=45.91.116.12:4258 , piece_id=KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:20.106010337 (601h42m19.977093372s) [noise=true , address=145.79.227.40:4053 , piece_id=JCSIUMWKEAQNQXTNSJTN6VSB3V2SIDRQK67QMK46B4GOUWEMQE2A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:20.106080911 (601h42m19.977163946s) [seg_pos={0 0}, upload=527370]: Segment reached optimal threshold of 14 pieces. Dec 30 17:11:20.106274520 (601h42m19.977357565s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Dec 30 17:11:20.106281644 (601h42m19.977364679s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold Dec 30 17:11:20.106351607 (601h42m19.977434642s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF Dec 30 17:11:20.106367767 (601h42m19.977450802s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold Dec 30 17:11:20.106424967 (601h42m19.977508002s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Dec 30 17:11:20.106432451 (601h42m19.977515486s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold Dec 30 17:11:20.106540666 (601h42m19.977623702s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 30 17:11:20.106546207 (601h42m19.977629242s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold Dec 30 17:11:20.106563380 (601h42m19.977646415s) [seg_pos={0 0}, upload=527370]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 17:11:20.106625529 (601h42m19.977708564s) [upload=527370]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 17:11:20.106629837 (601h42m19.977712882s) [upload=527370]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 17:11:20.406465452 (601h42m20.277548507s) [upload=527370]: Done uploading segments... Dec 30 17:11:20.406504206 (601h42m20.277587251s) [upload=527370]: Upload finished. err: <nil> Duration: 29.513500632s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [X....X.XX..X..X......X.........X..XXX...................X...] duration: 3.391709594s trace: https://jaeger.dev.storj.io/trace/14885d465b83b494 FAIL (2025-12-30 17:52:27 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.391734842s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [X.XXXXX.XX.X.XXX......XX.XXX...XX..X.....X.XXX....XX...XX.X.] duration: 6.1727348s trace: https://jaeger.dev.storj.io/trace/55a5575bf52571d FAIL (2025-12-30 17:52:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: O7PVMWJIAPR2K5B6KFLBXP6ZBSDSKLYKHDPK2KHAECUFUETLLWJA) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: HJCW5I7TEM6GUC3BJXVXBAHRD6RDIMP27FT42D5RIT7GJBJ5EI7A) context deadline exceeded ======LOGS====== 512 (602h22m35.143282547s) [upload=527968]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 17:51:35.526478113 (602h22m35.397561168s) [seg_pos={0 0}, upload=527968]: Begin upload segment... Dec 30 17:51:35.526532317 (602h22m35.397615362s) [seg_pos={0 0}, upload=527968]: Done begin upload segment. Dec 30 17:51:35.526556282 (602h22m35.397639318s) [noise=true , address=145.79.227.3:4150 , piece_id=HH63GEDKIMANNK5LDV3KJURKZLK4HS6XX6IBMIHLAR5TJOETLD3Q, seg_pos={0 0}, upload=527968]: Uploading piece... Dec 30 17:51:35.526599625 (602h22m35.39768267s) [noise=true , address=145.79.227.32:4834 , piece_id=HDWWSPRUY4UIBWUTL5ZSDWED6HRNUWVLV62YIITF36VI5BVQV6SA, seg_pos={0 0}, upload=527968]: Uploading piece... Dec 30 17:51:35.527118313 (602h22m35.398201358s) [noise=true , address=145.79.227.48:4302 , piece_id=W4ARNY5BMXG265ZPHR5ODXWG2PIML63M7LRPPAK5KXFJU6KFRIKA, seg_pos={0 0}, upload=527968]: Uploading piece... Dec 30 17:51:35.527481584 (602h22m35.398564629s) [noise=true , address=145.79.227.18:4007 , piece_id=FIL5NHZ4R43OOMRDEB6FGAUFNZFOX42KY2LOOABOEGJSSQ63NBGA, seg_pos={0 0}, upload=527968]: Uploading piece... Dec 30 17:51:35.527963932 (602h22m35.399046977s) [noise=true , address=145.79.227.11:4731 , piece_id=C2XTV32PSA2QE4ALLQUOQ5DV3MSFBXUAQS77662EWVKK7KN2DRCA, seg_pos={0 0}, upload=527968]: Uploading piece... Dec 30 17:51:35.528131381 (602h22m35.399214427s) [noise=true , address=77.90.31.13:4629 , piece_id=DSIJ2ZJ5PIJHGE3MLWAV3N5YXP4NSMYICCGQFC5IFCHDIAPTX5QQ, seg_pos={0 0}, upload=527968]: Uploading piece... Dec 30 17:51:35.528470187 (602h22m35.399553222s) [noise=true , address=145.79.227.26:4654 , piece_id=YWIM24OHCZXE6GSRHJRYVSHEHYOSDB44TRA2MU5FEO2TVXI3NLGQ, seg_pos={0 0}, upload=527968]: Uploading piece... Dec 30 17:51:35.528738076 (602h22m35.399821111s) [noise=true , address=45.91.116.13:4156 , piece_id=CGOHIJEJF6KEZJ2FQSLAZTBFZMCO5ZSOWPFP6M5S2AWMQN4PQG5A, seg_pos={0 0}, upload=527968]: Uploading piece... Dec 30 17:51:35.529078344 (602h22m35.400161379s) [noise=true , address=145.79.227.8:4330 , piece_id=5DI3WIVHWMSVOLWSJ2362LA67UV7ZS7ANBISLB4TOKAJHNJQYFBA, seg_pos={0 0}, upload=527968]: Uploading piece... Dec 30 17:51:35.529445825 (602h22m35.40052886s) [noise=true , address=145.79.227.45:4229 , piece_id=UTOX6A47KHU3CXHZYOA4UHGRKL4ZG3YQRI6WSP6VSCKIJVEKUTHA, seg_pos={0 0}, upload=527968]: Uploading piece... Dec 30 17:51:35.529931099 (602h22m35.401014144s) [noise=true , address=77.90.31.11:4556 , piece_id=ILBO6NM5YVZMFOE3A5NQYPHXJSYNCZPJPQLCRPCA7ZGIDMPL5I3Q, seg_pos={0 0}, upload=527968]: Uploading piece... Dec 30 17:51:35.530329828 (602h22m35.401412863s) [noise=true , address=145.79.227.5:4222 , piece_id=XTEWPRDP7ES2XHR5C3OJJOK5T2V44MVBQHR4WSG24CYNNFZSTFWQ, seg_pos={0 0}, upload=527968]: Uploading piece... Dec 30 17:51:35.535931695 (602h22m35.40701473s) [noise=true , address=145.79.227.7:4294 , piece_id=HJTGM4NMZBTGHYOLGNQEGDYA5OOSW4SBAAVVULOEQNIC2MMYPDJA, seg_pos={0 0}, upload=527968]: Uploading piece... Dec 30 17:51:35.538912971 (602h22m35.409996006s) [noise=true , address=45.91.116.5:4300 , piece_id=O7PVMWJIAPR2K5B6KFLBXP6ZBSDSKLYKHDPK2KHAECUFUETLLWJA, seg_pos={0 0}, upload=527968]: Uploading piece... Dec 30 17:51:35.540939429 (602h22m35.412022474s) [noise=true , address=145.79.227.41:4087 , piece_id=C6JPQC7QDPPR5A3EWGMKX564RANGK6K5XAZ564PNW6VNTGCYMDJA, seg_pos={0 0}, upload=527968]: Uploading piece... Dec 30 17:51:35.542806936 (602h22m35.413889971s) [noise=true , address=45.91.116.14:4191 , piece_id=HJCW5I7TEM6GUC3BJXVXBAHRD6RDIMP27FT42D5RIT7GJBJ5EI7A, seg_pos={0 0}, upload=527968]: Uploading piece... Dec 30 17:51:37.224837605 (602h22m37.09592065s) [upload=527968]: Next returned nil segment Dec 30 17:51:37.224856240 (602h22m37.095939275s) [upload=527968]: Waiting for error group managing segments... Dec 30 17:51:37.495619832 (602h22m37.366702867s) [noise=true , address=145.79.227.11:4731 , piece_id=C2XTV32PSA2QE4ALLQUOQ5DV3MSFBXUAQS77662EWVKK7KN2DRCA, seg_pos={0 0}, upload=527968]: Done uploading piece. err: <nil> Dec 30 17:51:37.524065847 (602h22m37.395148882s) [noise=true , address=145.79.227.3:4150 , piece_id=HH63GEDKIMANNK5LDV3KJURKZLK4HS6XX6IBMIHLAR5TJOETLD3Q, seg_pos={0 0}, upload=527968]: Done uploading piece. err: <nil> Dec 30 17:51:37.534746695 (602h22m37.40582974s) [noise=true , address=145.79.227.7:4294 , piece_id=HJTGM4NMZBTGHYOLGNQEGDYA5OOSW4SBAAVVULOEQNIC2MMYPDJA, seg_pos={0 0}, upload=527968]: Done uploading piece. err: <nil> Dec 30 17:51:37.540151817 (602h22m37.411234862s) [noise=true , address=145.79.227.18:4007 , piece_id=FIL5NHZ4R43OOMRDEB6FGAUFNZFOX42KY2LOOABOEGJSSQ63NBGA, seg_pos={0 0}, upload=527968]: Done uploading piece. err: <nil> Dec 30 17:51:37.545407774 (602h22m37.416490809s) [noise=true , address=145.79.227.32:4834 , piece_id=HDWWSPRUY4UIBWUTL5ZSDWED6HRNUWVLV62YIITF36VI5BVQV6SA, seg_pos={0 0}, upload=527968]: Done uploading piece. err: <nil> Dec 30 17:51:37.547712634 (602h22m37.418795669s) [noise=true , address=145.79.227.41:4087 , piece_id=C6JPQC7QDPPR5A3EWGMKX564RANGK6K5XAZ564PNW6VNTGCYMDJA, seg_pos={0 0}, upload=527968]: Done uploading piece. err: <nil> Dec 30 17:51:37.614567254 (602h22m37.485650299s) [noise=true , address=145.79.227.45:4229 , piece_id=UTOX6A47KHU3CXHZYOA4UHGRKL4ZG3YQRI6WSP6VSCKIJVEKUTHA, seg_pos={0 0}, upload=527968]: Done uploading piece. err: <nil> Dec 30 17:51:37.690150225 (602h22m37.56123327s) [noise=true , address=145.79.227.48:4302 , piece_id=W4ARNY5BMXG265ZPHR5ODXWG2PIML63M7LRPPAK5KXFJU6KFRIKA, seg_pos={0 0}, upload=527968]: Done uploading piece. err: <nil> Dec 30 17:51:37.700023094 (602h22m37.571106159s) [noise=true , address=145.79.227.8:4330 , piece_id=5DI3WIVHWMSVOLWSJ2362LA67UV7ZS7ANBISLB4TOKAJHNJQYFBA, seg_pos={0 0}, upload=527968]: Done uploading piece. err: <nil> Dec 30 17:51:37.711035393 (602h22m37.582118428s) [noise=true , address=77.90.31.13:4629 , piece_id=DSIJ2ZJ5PIJHGE3MLWAV3N5YXP4NSMYICCGQFC5IFCHDIAPTX5QQ, seg_pos={0 0}, upload=527968]: Done uploading piece. err: <nil> Dec 30 17:51:37.823628987 (602h22m37.694712022s) [noise=true , address=77.90.31.11:4556 , piece_id=ILBO6NM5YVZMFOE3A5NQYPHXJSYNCZPJPQLCRPCA7ZGIDMPL5I3Q, seg_pos={0 0}, upload=527968]: Done uploading piece. err: <nil> Dec 30 17:51:37.897500668 (602h22m37.768583704s) [noise=true , address=45.91.116.5:4300 , piece_id=O7PVMWJIAPR2K5B6KFLBXP6ZBSDSKLYKHDPK2KHAECUFUETLLWJA, seg_pos={0 0}, upload=527968]: Done uploading piece. err: <nil> Dec 30 17:51:38.673987997 (602h22m38.545071033s) [noise=true , address=45.91.116.14:4191 , piece_id=HJCW5I7TEM6GUC3BJXVXBAHRD6RDIMP27FT42D5RIT7GJBJ5EI7A, seg_pos={0 0}, upload=527968]: Done uploading piece. err: <nil> Dec 30 17:51:38.674011743 (602h22m38.545094788s) [seg_pos={0 0}, upload=527968]: Segment reached optimal threshold of 13 pieces. Dec 30 17:51:38.674214909 (602h22m38.545297944s) [noise=true , address=45.91.116.13:4156 , piece_id=CGOHIJEJF6KEZJ2FQSLAZTBFZMCO5ZSOWPFP6M5S2AWMQN4PQG5A, seg_pos={0 0}, upload=527968]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:57938->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:57938->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57938->45.91.116.13:4156: use of closed network connection Dec 30 17:51:38.674225169 (602h22m38.545308204s) [noise=true , address=45.91.116.13:4156 , piece_id=CGOHIJEJF6KEZJ2FQSLAZTBFZMCO5ZSOWPFP6M5S2AWMQN4PQG5A, seg_pos={0 0}, upload=527968]: Long tail cancellation due to optimal threshold Dec 30 17:51:38.674236881 (602h22m38.545319926s) [noise=true , address=145.79.227.26:4654 , piece_id=YWIM24OHCZXE6GSRHJRYVSHEHYOSDB44TRA2MU5FEO2TVXI3NLGQ, seg_pos={0 0}, upload=527968]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: write tcp 188.166.225.170:53476->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:53476->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53476->145.79.227.26:4654: use of closed network connection Dec 30 17:51:38.674248904 (602h22m38.545331939s) [noise=true , address=145.79.227.26:4654 , piece_id=YWIM24OHCZXE6GSRHJRYVSHEHYOSDB44TRA2MU5FEO2TVXI3NLGQ, seg_pos={0 0}, upload=527968]: Long tail cancellation due to optimal threshold Dec 30 17:51:38.674556309 (602h22m38.545639344s) [noise=true , address=145.79.227.5:4222 , piece_id=XTEWPRDP7ES2XHR5C3OJJOK5T2V44MVBQHR4WSG24CYNNFZSTFWQ, seg_pos={0 0}, upload=527968]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: write tcp 188.166.225.170:58382->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:58382->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58382->145.79.227.5:4222: use of closed network connection Dec 30 17:51:38.674566409 (602h22m38.545649454s) [noise=true , address=145.79.227.5:4222 , piece_id=XTEWPRDP7ES2XHR5C3OJJOK5T2V44MVBQHR4WSG24CYNNFZSTFWQ, seg_pos={0 0}, upload=527968]: Long tail cancellation due to optimal threshold Dec 30 17:51:38.674588350 (602h22m38.545671395s) [seg_pos={0 0}, upload=527968]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 17:51:38.674670117 (602h22m38.545753152s) [upload=527968]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 17:51:38.674676058 (602h22m38.545759103s) [upload=527968]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 17:51:38.973136859 (602h22m38.844219894s) [upload=527968]: Done uploading segments... Dec 30 17:51:38.973157288 (602h22m38.844240333s) [upload=527968]: Upload finished. err: <nil> Duration:* 6.172752483s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [.X.XXXXX.XX.X.XXX......XX.XXX...XX..X.....X.XXX....XX...XX.X] duration: 30.004286458s trace: https://jaeger.dev.storj.io/trace/55a5575bf52571d FAIL (2025-12-30 17:52:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: O7PVMWJIAPR2K5B6KFLBXP6ZBSDSKLYKHDPK2KHAECUFUETLLWJA) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: HJCW5I7TEM6GUC3BJXVXBAHRD6RDIMP27FT42D5RIT7GJBJ5EI7A) context deadline exceeded ======LOGS====== 512 (602h22m35.143282547s) [upload=527968]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 17:51:35.526478113 (602h22m35.397561168s) [seg_pos={0 0}, upload=527968]: Begin upload segment... Dec 30 17:51:35.526532317 (602h22m35.397615362s) [seg_pos={0 0}, upload=527968]: Done begin upload segment. Dec 30 17:51:35.526556282 (602h22m35.397639318s) [noise=true , address=145.79.227.3:4150 , piece_id=HH63GEDKIMANNK5LDV3KJURKZLK4HS6XX6IBMIHLAR5TJOETLD3Q, seg_pos={0 0}, upload=527968]: Uploading piece... Dec 30 17:51:35.526599625 (602h22m35.39768267s) [noise=true , address=145.79.227.32:4834 , piece_id=HDWWSPRUY4UIBWUTL5ZSDWED6HRNUWVLV62YIITF36VI5BVQV6SA, seg_pos={0 0}, upload=527968]: Uploading piece... Dec 30 17:51:35.527118313 (602h22m35.398201358s) [noise=true , address=145.79.227.48:4302 , piece_id=W4ARNY5BMXG265ZPHR5ODXWG2PIML63M7LRPPAK5KXFJU6KFRIKA, seg_pos={0 0}, upload=527968]: Uploading piece... Dec 30 17:51:35.527481584 (602h22m35.398564629s) [noise=true , address=145.79.227.18:4007 , piece_id=FIL5NHZ4R43OOMRDEB6FGAUFNZFOX42KY2LOOABOEGJSSQ63NBGA, seg_pos={0 0}, upload=527968]: Uploading piece... Dec 30 17:51:35.527963932 (602h22m35.399046977s) [noise=true , address=145.79.227.11:4731 , piece_id=C2XTV32PSA2QE4ALLQUOQ5DV3MSFBXUAQS77662EWVKK7KN2DRCA, seg_pos={0 0}, upload=527968]: Uploading piece... Dec 30 17:51:35.528131381 (602h22m35.399214427s) [noise=true , address=77.90.31.13:4629 , piece_id=DSIJ2ZJ5PIJHGE3MLWAV3N5YXP4NSMYICCGQFC5IFCHDIAPTX5QQ, seg_pos={0 0}, upload=527968]: Uploading piece... Dec 30 17:51:35.528470187 (602h22m35.399553222s) [noise=true , address=145.79.227.26:4654 , piece_id=YWIM24OHCZXE6GSRHJRYVSHEHYOSDB44TRA2MU5FEO2TVXI3NLGQ, seg_pos={0 0}, upload=527968]: Uploading piece... Dec 30 17:51:35.528738076 (602h22m35.399821111s) [noise=true , address=45.91.116.13:4156 , piece_id=CGOHIJEJF6KEZJ2FQSLAZTBFZMCO5ZSOWPFP6M5S2AWMQN4PQG5A, seg_pos={0 0}, upload=527968]: Uploading piece... Dec 30 17:51:35.529078344 (602h22m35.400161379s) [noise=true , address=145.79.227.8:4330 , piece_id=5DI3WIVHWMSVOLWSJ2362LA67UV7ZS7ANBISLB4TOKAJHNJQYFBA, seg_pos={0 0}, upload=527968]: Uploading piece... Dec 30 17:51:35.529445825 (602h22m35.40052886s) [noise=true , address=145.79.227.45:4229 , piece_id=UTOX6A47KHU3CXHZYOA4UHGRKL4ZG3YQRI6WSP6VSCKIJVEKUTHA, seg_pos={0 0}, upload=527968]: Uploading piece... Dec 30 17:51:35.529931099 (602h22m35.401014144s) [noise=true , address=77.90.31.11:4556 , piece_id=ILBO6NM5YVZMFOE3A5NQYPHXJSYNCZPJPQLCRPCA7ZGIDMPL5I3Q, seg_pos={0 0}, upload=527968]: Uploading piece... Dec 30 17:51:35.530329828 (602h22m35.401412863s) [noise=true , address=145.79.227.5:4222 , piece_id=XTEWPRDP7ES2XHR5C3OJJOK5T2V44MVBQHR4WSG24CYNNFZSTFWQ, seg_pos={0 0}, upload=527968]: Uploading piece... Dec 30 17:51:35.535931695 (602h22m35.40701473s) [noise=true , address=145.79.227.7:4294 , piece_id=HJTGM4NMZBTGHYOLGNQEGDYA5OOSW4SBAAVVULOEQNIC2MMYPDJA, seg_pos={0 0}, upload=527968]: Uploading piece... Dec 30 17:51:35.538912971 (602h22m35.409996006s) [noise=true , address=45.91.116.5:4300 , piece_id=O7PVMWJIAPR2K5B6KFLBXP6ZBSDSKLYKHDPK2KHAECUFUETLLWJA, seg_pos={0 0}, upload=527968]: Uploading piece... Dec 30 17:51:35.540939429 (602h22m35.412022474s) [noise=true , address=145.79.227.41:4087 , piece_id=C6JPQC7QDPPR5A3EWGMKX564RANGK6K5XAZ564PNW6VNTGCYMDJA, seg_pos={0 0}, upload=527968]: Uploading piece... Dec 30 17:51:35.542806936 (602h22m35.413889971s) [noise=true , address=45.91.116.14:4191 , piece_id=HJCW5I7TEM6GUC3BJXVXBAHRD6RDIMP27FT42D5RIT7GJBJ5EI7A, seg_pos={0 0}, upload=527968]: Uploading piece... Dec 30 17:51:37.224837605 (602h22m37.09592065s) [upload=527968]: Next returned nil segment Dec 30 17:51:37.224856240 (602h22m37.095939275s) [upload=527968]: Waiting for error group managing segments... Dec 30 17:51:37.495619832 (602h22m37.366702867s) [noise=true , address=145.79.227.11:4731 , piece_id=C2XTV32PSA2QE4ALLQUOQ5DV3MSFBXUAQS77662EWVKK7KN2DRCA, seg_pos={0 0}, upload=527968]: Done uploading piece. err: <nil> Dec 30 17:51:37.524065847 (602h22m37.395148882s) [noise=true , address=145.79.227.3:4150 , piece_id=HH63GEDKIMANNK5LDV3KJURKZLK4HS6XX6IBMIHLAR5TJOETLD3Q, seg_pos={0 0}, upload=527968]: Done uploading piece. err: <nil> Dec 30 17:51:37.534746695 (602h22m37.40582974s) [noise=true , address=145.79.227.7:4294 , piece_id=HJTGM4NMZBTGHYOLGNQEGDYA5OOSW4SBAAVVULOEQNIC2MMYPDJA, seg_pos={0 0}, upload=527968]: Done uploading piece. err: <nil> Dec 30 17:51:37.540151817 (602h22m37.411234862s) [noise=true , address=145.79.227.18:4007 , piece_id=FIL5NHZ4R43OOMRDEB6FGAUFNZFOX42KY2LOOABOEGJSSQ63NBGA, seg_pos={0 0}, upload=527968]: Done uploading piece. err: <nil> Dec 30 17:51:37.545407774 (602h22m37.416490809s) [noise=true , address=145.79.227.32:4834 , piece_id=HDWWSPRUY4UIBWUTL5ZSDWED6HRNUWVLV62YIITF36VI5BVQV6SA, seg_pos={0 0}, upload=527968]: Done uploading piece. err: <nil> Dec 30 17:51:37.547712634 (602h22m37.418795669s) [noise=true , address=145.79.227.41:4087 , piece_id=C6JPQC7QDPPR5A3EWGMKX564RANGK6K5XAZ564PNW6VNTGCYMDJA, seg_pos={0 0}, upload=527968]: Done uploading piece. err: <nil> Dec 30 17:51:37.614567254 (602h22m37.485650299s) [noise=true , address=145.79.227.45:4229 , piece_id=UTOX6A47KHU3CXHZYOA4UHGRKL4ZG3YQRI6WSP6VSCKIJVEKUTHA, seg_pos={0 0}, upload=527968]: Done uploading piece. err: <nil> Dec 30 17:51:37.690150225 (602h22m37.56123327s) [noise=true , address=145.79.227.48:4302 , piece_id=W4ARNY5BMXG265ZPHR5ODXWG2PIML63M7LRPPAK5KXFJU6KFRIKA, seg_pos={0 0}, upload=527968]: Done uploading piece. err: <nil> Dec 30 17:51:37.700023094 (602h22m37.571106159s) [noise=true , address=145.79.227.8:4330 , piece_id=5DI3WIVHWMSVOLWSJ2362LA67UV7ZS7ANBISLB4TOKAJHNJQYFBA, seg_pos={0 0}, upload=527968]: Done uploading piece. err: <nil> Dec 30 17:51:37.711035393 (602h22m37.582118428s) [noise=true , address=77.90.31.13:4629 , piece_id=DSIJ2ZJ5PIJHGE3MLWAV3N5YXP4NSMYICCGQFC5IFCHDIAPTX5QQ, seg_pos={0 0}, upload=527968]: Done uploading piece. err: <nil> Dec 30 17:51:37.823628987 (602h22m37.694712022s) [noise=true , address=77.90.31.11:4556 , piece_id=ILBO6NM5YVZMFOE3A5NQYPHXJSYNCZPJPQLCRPCA7ZGIDMPL5I3Q, seg_pos={0 0}, upload=527968]: Done uploading piece. err: <nil> Dec 30 17:51:37.897500668 (602h22m37.768583704s) [noise=true , address=45.91.116.5:4300 , piece_id=O7PVMWJIAPR2K5B6KFLBXP6ZBSDSKLYKHDPK2KHAECUFUETLLWJA, seg_pos={0 0}, upload=527968]: Done uploading piece. err: <nil> Dec 30 17:51:38.673987997 (602h22m38.545071033s) [noise=true , address=45.91.116.14:4191 , piece_id=HJCW5I7TEM6GUC3BJXVXBAHRD6RDIMP27FT42D5RIT7GJBJ5EI7A, seg_pos={0 0}, upload=527968]: Done uploading piece. err: <nil> Dec 30 17:51:38.674011743 (602h22m38.545094788s) [seg_pos={0 0}, upload=527968]: Segment reached optimal threshold of 13 pieces. Dec 30 17:51:38.674214909 (602h22m38.545297944s) [noise=true , address=45.91.116.13:4156 , piece_id=CGOHIJEJF6KEZJ2FQSLAZTBFZMCO5ZSOWPFP6M5S2AWMQN4PQG5A, seg_pos={0 0}, upload=527968]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy): protocol: write tcp 188.166.225.170:57938->45.91.116.13:4156: use of closed network connection; write tcp 188.166.225.170:57938->45.91.116.13:4156: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57938->45.91.116.13:4156: use of closed network connection Dec 30 17:51:38.674225169 (602h22m38.545308204s) [noise=true , address=45.91.116.13:4156 , piece_id=CGOHIJEJF6KEZJ2FQSLAZTBFZMCO5ZSOWPFP6M5S2AWMQN4PQG5A, seg_pos={0 0}, upload=527968]: Long tail cancellation due to optimal threshold Dec 30 17:51:38.674236881 (602h22m38.545319926s) [noise=true , address=145.79.227.26:4654 , piece_id=YWIM24OHCZXE6GSRHJRYVSHEHYOSDB44TRA2MU5FEO2TVXI3NLGQ, seg_pos={0 0}, upload=527968]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: write tcp 188.166.225.170:53476->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:53476->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53476->145.79.227.26:4654: use of closed network connection Dec 30 17:51:38.674248904 (602h22m38.545331939s) [noise=true , address=145.79.227.26:4654 , piece_id=YWIM24OHCZXE6GSRHJRYVSHEHYOSDB44TRA2MU5FEO2TVXI3NLGQ, seg_pos={0 0}, upload=527968]: Long tail cancellation due to optimal threshold Dec 30 17:51:38.674556309 (602h22m38.545639344s) [noise=true , address=145.79.227.5:4222 , piece_id=XTEWPRDP7ES2XHR5C3OJJOK5T2V44MVBQHR4WSG24CYNNFZSTFWQ, seg_pos={0 0}, upload=527968]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1): protocol: write tcp 188.166.225.170:58382->145.79.227.5:4222: use of closed network connection; write tcp 188.166.225.170:58382->145.79.227.5:4222: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58382->145.79.227.5:4222: use of closed network connection Dec 30 17:51:38.674566409 (602h22m38.545649454s) [noise=true , address=145.79.227.5:4222 , piece_id=XTEWPRDP7ES2XHR5C3OJJOK5T2V44MVBQHR4WSG24CYNNFZSTFWQ, seg_pos={0 0}, upload=527968]: Long tail cancellation due to optimal threshold Dec 30 17:51:38.674588350 (602h22m38.545671395s) [seg_pos={0 0}, upload=527968]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 17:51:38.674670117 (602h22m38.545753152s) [upload=527968]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 17:51:38.674676058 (602h22m38.545759103s) [upload=527968]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 17:51:38.973136859 (602h22m38.844219894s) [upload=527968]: Done uploading segments... Dec 30 17:51:38.973157288 (602h22m38.844240333s) [upload=527968]: Upload finished. err: <nil> Duration:* 30.00430901s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 15.55000854s trace: https://jaeger.dev.storj.io/trace/5ea8d7d4f61e13a2 FAIL (2025-12-30 17:52:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1X5A7bLqMEQuaMbf9xZ7PDUD4161sz2QqdM9r59y3bsyPBJhaK, Piece ID: HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ) context deadline exceeded; piecestore: (Node ID: 12eLKUg2vTnZMrLaraSPEoGXe3jDVBJYyURdcn4f2gCN5pvvCQ8, Piece ID: KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ) context deadline exceeded; piecestore: (Node ID: 12W4n6uf6azQk1QgeR8Xw7UkpnZm19zdyFKwojCsqZt2GFHrpce, Piece ID: DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ) context deadline exceeded ======LOGS====== 17.114181822 (601h42m16.985264867s) [noise=true , address=145.79.227.44:4208 , piece_id=3OXDCHZSOIFS5VQCI4SXSAQFICDCYHRE5NGFBERR33KOTGSZWLDQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.114641387 (601h42m16.985724432s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.115652532 (601h42m16.986735578s) [noise=true , address=77.90.31.17:4790 , piece_id=5RSRESKGDA2YF4NBRF4ZFABBKWJK4NO6VLEXW2Z6EY3OPXH2WPNA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.115937134 (601h42m16.987020179s) [noise=true , address=64.49.13.155:4923 , piece_id=MUJD7XLM2YRLRCHYVSDPBE6VBYLHCOA7HJQP22ZSET4N574XQMJA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.116814986 (601h42m16.987898031s) [noise=true , address=45.91.116.11:4139 , piece_id=72DPEWZ7ZPZZ53UGYV2VCUKLCK7HCM22CDBRHWVLJGDGO5B5JADQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.117501182 (601h42m16.988584228s) [noise=true , address=77.90.31.14:4665 , piece_id=DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.118263334 (601h42m16.989346379s) [noise=true , address=145.79.227.40:4084 , piece_id=ZADZDRKBIILLAI4IT335CYETOSFOFUUPQIH6HZJCTHKRTZSPDJUQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.118555850 (601h42m16.989638886s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.118765930 (601h42m16.989849036s) [noise=true , address=77.90.31.21:4905 , piece_id=HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.119029533 (601h42m16.990112568s) [noise=true , address=77.90.31.10:4533 , piece_id=QFBBYNQHQL2G7PEREK43FZBRUMKMVNBC2XKTE2NWTVTKN2452LPA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.119417672 (601h42m16.990500707s) [noise=true , address=45.91.116.12:4258 , piece_id=KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.119441588 (601h42m16.990524643s) [noise=true , address=145.79.227.40:4053 , piece_id=JCSIUMWKEAQNQXTNSJTN6VSB3V2SIDRQK67QMK46B4GOUWEMQE2A, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.119763470 (601h42m16.990846505s) [noise=true , address=77.90.31.18:4821 , piece_id=MXZFKXM3JV2HQURDLIU7BKWLAKDZ27O4ER3SGC7QBZV5I3J2EHKA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.120028685 (601h42m16.99111172s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.120086986 (601h42m16.991170031s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.120586507 (601h42m16.991669562s) [noise=true , address=45.91.116.13:4168 , piece_id=AKUE5H3A2UECQOZY3GV3MWZKIHSNKCDOG32DKCNHIRTJROJ74M3A, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.121258768 (601h42m16.992341813s) [noise=true , address=145.79.227.9:4379 , piece_id=QTFXFSGGKYBQSPWJYBBJMQUL3D5EEO3IX4C53ECD7FXMUOGUBC7Q, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:18.568739165 (601h42m18.43982221s) [upload=527370]: Next returned nil segment Dec 30 17:11:18.568751328 (601h42m18.439834363s) [upload=527370]: Waiting for error group managing segments... Dec 30 17:11:18.978424178 (601h42m18.849507213s) [noise=true , address=64.49.13.155:4923 , piece_id=MUJD7XLM2YRLRCHYVSDPBE6VBYLHCOA7HJQP22ZSET4N574XQMJA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:18.986258125 (601h42m18.85734116s) [noise=true , address=145.79.227.9:4379 , piece_id=QTFXFSGGKYBQSPWJYBBJMQUL3D5EEO3IX4C53ECD7FXMUOGUBC7Q, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.274519421 (601h42m19.145602456s) [noise=true , address=77.90.31.14:4665 , piece_id=DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.315775438 (601h42m19.186858473s) [noise=true , address=45.91.116.13:4168 , piece_id=AKUE5H3A2UECQOZY3GV3MWZKIHSNKCDOG32DKCNHIRTJROJ74M3A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.335765371 (601h42m19.206848406s) [noise=true , address=77.90.31.21:4905 , piece_id=HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.354812129 (601h42m19.225895164s) [noise=true , address=77.90.31.10:4533 , piece_id=QFBBYNQHQL2G7PEREK43FZBRUMKMVNBC2XKTE2NWTVTKN2452LPA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.392024917 (601h42m19.263107952s) [noise=true , address=45.91.116.11:4139 , piece_id=72DPEWZ7ZPZZ53UGYV2VCUKLCK7HCM22CDBRHWVLJGDGO5B5JADQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.396084327 (601h42m19.267167372s) [noise=true , address=77.90.31.18:4821 , piece_id=MXZFKXM3JV2HQURDLIU7BKWLAKDZ27O4ER3SGC7QBZV5I3J2EHKA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.862099099 (601h42m19.733182134s) [noise=true , address=77.90.31.17:4790 , piece_id=5RSRESKGDA2YF4NBRF4ZFABBKWJK4NO6VLEXW2Z6EY3OPXH2WPNA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.905412913 (601h42m19.776495948s) [noise=true , address=145.79.227.44:4208 , piece_id=3OXDCHZSOIFS5VQCI4SXSAQFICDCYHRE5NGFBERR33KOTGSZWLDQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.910309688 (601h42m19.781392743s) [noise=true , address=145.79.227.30:4794 , piece_id=EJH6Q3T2BMPVLG6F7CW7N4GYOA3EZFPADL6VPT7WKKG65B2WZVXQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.934233361 (601h42m19.805316397s) [noise=true , address=145.79.227.40:4084 , piece_id=ZADZDRKBIILLAI4IT335CYETOSFOFUUPQIH6HZJCTHKRTZSPDJUQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:20.074011527 (601h42m19.945094572s) [noise=true , address=45.91.116.12:4258 , piece_id=KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:20.106010337 (601h42m19.977093372s) [noise=true , address=145.79.227.40:4053 , piece_id=JCSIUMWKEAQNQXTNSJTN6VSB3V2SIDRQK67QMK46B4GOUWEMQE2A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:20.106080911 (601h42m19.977163946s) [seg_pos={0 0}, upload=527370]: Segment reached optimal threshold of 14 pieces. Dec 30 17:11:20.106274520 (601h42m19.977357565s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Dec 30 17:11:20.106281644 (601h42m19.977364679s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold Dec 30 17:11:20.106351607 (601h42m19.977434642s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF Dec 30 17:11:20.106367767 (601h42m19.977450802s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold Dec 30 17:11:20.106424967 (601h42m19.977508002s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Dec 30 17:11:20.106432451 (601h42m19.977515486s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold Dec 30 17:11:20.106540666 (601h42m19.977623702s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 30 17:11:20.106546207 (601h42m19.977629242s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold Dec 30 17:11:20.106563380 (601h42m19.977646415s) [seg_pos={0 0}, upload=527370]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 17:11:20.106625529 (601h42m19.977708564s) [upload=527370]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 17:11:20.106629837 (601h42m19.977712882s) [upload=527370]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 17:11:20.406465452 (601h42m20.277548507s) [upload=527370]: Done uploading segments... Dec 30 17:11:20.406504206 (601h42m20.277587251s) [upload=527370]: Upload finished. err: <nil> Duration: 15.550037175s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X...X..X............................XX......................] duration: 5.810232817s trace: https://jaeger.dev.storj.io/trace/4173ea2fcd946b89 FAIL (2025-12-30 17:51:52 UTC) orbiter warning: context deadline exceeded Duration: 5.810262043s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 40.133274667s trace: https://jaeger.dev.storj.io/trace/5ea8d7d4f61e13a2 FAIL (2025-12-30 17:51:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1X5A7bLqMEQuaMbf9xZ7PDUD4161sz2QqdM9r59y3bsyPBJhaK, Piece ID: HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ) context deadline exceeded; piecestore: (Node ID: 12eLKUg2vTnZMrLaraSPEoGXe3jDVBJYyURdcn4f2gCN5pvvCQ8, Piece ID: KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ) context deadline exceeded; piecestore: (Node ID: 12W4n6uf6azQk1QgeR8Xw7UkpnZm19zdyFKwojCsqZt2GFHrpce, Piece ID: DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ) context deadline exceeded ======LOGS====== 17.114181822 (601h42m16.985264867s) [noise=true , address=145.79.227.44:4208 , piece_id=3OXDCHZSOIFS5VQCI4SXSAQFICDCYHRE5NGFBERR33KOTGSZWLDQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.114641387 (601h42m16.985724432s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.115652532 (601h42m16.986735578s) [noise=true , address=77.90.31.17:4790 , piece_id=5RSRESKGDA2YF4NBRF4ZFABBKWJK4NO6VLEXW2Z6EY3OPXH2WPNA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.115937134 (601h42m16.987020179s) [noise=true , address=64.49.13.155:4923 , piece_id=MUJD7XLM2YRLRCHYVSDPBE6VBYLHCOA7HJQP22ZSET4N574XQMJA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.116814986 (601h42m16.987898031s) [noise=true , address=45.91.116.11:4139 , piece_id=72DPEWZ7ZPZZ53UGYV2VCUKLCK7HCM22CDBRHWVLJGDGO5B5JADQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.117501182 (601h42m16.988584228s) [noise=true , address=77.90.31.14:4665 , piece_id=DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.118263334 (601h42m16.989346379s) [noise=true , address=145.79.227.40:4084 , piece_id=ZADZDRKBIILLAI4IT335CYETOSFOFUUPQIH6HZJCTHKRTZSPDJUQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.118555850 (601h42m16.989638886s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.118765930 (601h42m16.989849036s) [noise=true , address=77.90.31.21:4905 , piece_id=HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.119029533 (601h42m16.990112568s) [noise=true , address=77.90.31.10:4533 , piece_id=QFBBYNQHQL2G7PEREK43FZBRUMKMVNBC2XKTE2NWTVTKN2452LPA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.119417672 (601h42m16.990500707s) [noise=true , address=45.91.116.12:4258 , piece_id=KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.119441588 (601h42m16.990524643s) [noise=true , address=145.79.227.40:4053 , piece_id=JCSIUMWKEAQNQXTNSJTN6VSB3V2SIDRQK67QMK46B4GOUWEMQE2A, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.119763470 (601h42m16.990846505s) [noise=true , address=77.90.31.18:4821 , piece_id=MXZFKXM3JV2HQURDLIU7BKWLAKDZ27O4ER3SGC7QBZV5I3J2EHKA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.120028685 (601h42m16.99111172s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.120086986 (601h42m16.991170031s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.120586507 (601h42m16.991669562s) [noise=true , address=45.91.116.13:4168 , piece_id=AKUE5H3A2UECQOZY3GV3MWZKIHSNKCDOG32DKCNHIRTJROJ74M3A, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.121258768 (601h42m16.992341813s) [noise=true , address=145.79.227.9:4379 , piece_id=QTFXFSGGKYBQSPWJYBBJMQUL3D5EEO3IX4C53ECD7FXMUOGUBC7Q, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:18.568739165 (601h42m18.43982221s) [upload=527370]: Next returned nil segment Dec 30 17:11:18.568751328 (601h42m18.439834363s) [upload=527370]: Waiting for error group managing segments... Dec 30 17:11:18.978424178 (601h42m18.849507213s) [noise=true , address=64.49.13.155:4923 , piece_id=MUJD7XLM2YRLRCHYVSDPBE6VBYLHCOA7HJQP22ZSET4N574XQMJA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:18.986258125 (601h42m18.85734116s) [noise=true , address=145.79.227.9:4379 , piece_id=QTFXFSGGKYBQSPWJYBBJMQUL3D5EEO3IX4C53ECD7FXMUOGUBC7Q, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.274519421 (601h42m19.145602456s) [noise=true , address=77.90.31.14:4665 , piece_id=DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.315775438 (601h42m19.186858473s) [noise=true , address=45.91.116.13:4168 , piece_id=AKUE5H3A2UECQOZY3GV3MWZKIHSNKCDOG32DKCNHIRTJROJ74M3A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.335765371 (601h42m19.206848406s) [noise=true , address=77.90.31.21:4905 , piece_id=HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.354812129 (601h42m19.225895164s) [noise=true , address=77.90.31.10:4533 , piece_id=QFBBYNQHQL2G7PEREK43FZBRUMKMVNBC2XKTE2NWTVTKN2452LPA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.392024917 (601h42m19.263107952s) [noise=true , address=45.91.116.11:4139 , piece_id=72DPEWZ7ZPZZ53UGYV2VCUKLCK7HCM22CDBRHWVLJGDGO5B5JADQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.396084327 (601h42m19.267167372s) [noise=true , address=77.90.31.18:4821 , piece_id=MXZFKXM3JV2HQURDLIU7BKWLAKDZ27O4ER3SGC7QBZV5I3J2EHKA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.862099099 (601h42m19.733182134s) [noise=true , address=77.90.31.17:4790 , piece_id=5RSRESKGDA2YF4NBRF4ZFABBKWJK4NO6VLEXW2Z6EY3OPXH2WPNA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.905412913 (601h42m19.776495948s) [noise=true , address=145.79.227.44:4208 , piece_id=3OXDCHZSOIFS5VQCI4SXSAQFICDCYHRE5NGFBERR33KOTGSZWLDQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.910309688 (601h42m19.781392743s) [noise=true , address=145.79.227.30:4794 , piece_id=EJH6Q3T2BMPVLG6F7CW7N4GYOA3EZFPADL6VPT7WKKG65B2WZVXQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.934233361 (601h42m19.805316397s) [noise=true , address=145.79.227.40:4084 , piece_id=ZADZDRKBIILLAI4IT335CYETOSFOFUUPQIH6HZJCTHKRTZSPDJUQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:20.074011527 (601h42m19.945094572s) [noise=true , address=45.91.116.12:4258 , piece_id=KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:20.106010337 (601h42m19.977093372s) [noise=true , address=145.79.227.40:4053 , piece_id=JCSIUMWKEAQNQXTNSJTN6VSB3V2SIDRQK67QMK46B4GOUWEMQE2A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:20.106080911 (601h42m19.977163946s) [seg_pos={0 0}, upload=527370]: Segment reached optimal threshold of 14 pieces. Dec 30 17:11:20.106274520 (601h42m19.977357565s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Dec 30 17:11:20.106281644 (601h42m19.977364679s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold Dec 30 17:11:20.106351607 (601h42m19.977434642s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF Dec 30 17:11:20.106367767 (601h42m19.977450802s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold Dec 30 17:11:20.106424967 (601h42m19.977508002s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Dec 30 17:11:20.106432451 (601h42m19.977515486s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold Dec 30 17:11:20.106540666 (601h42m19.977623702s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 30 17:11:20.106546207 (601h42m19.977629242s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold Dec 30 17:11:20.106563380 (601h42m19.977646415s) [seg_pos={0 0}, upload=527370]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 17:11:20.106625529 (601h42m19.977708564s) [upload=527370]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 17:11:20.106629837 (601h42m19.977712882s) [upload=527370]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 17:11:20.406465452 (601h42m20.277548507s) [upload=527370]: Done uploading segments... Dec 30 17:11:20.406504206 (601h42m20.277587251s) [upload=527370]: Upload finished. err: <nil> Duration: 40.133305475s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [.X....X.XX..X..X......X.........X..XXX...................X..] duration: 3.720466564s trace: https://jaeger.dev.storj.io/trace/14885d465b83b494 FAIL (2025-12-30 17:51:14 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.720493836s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X...X..X............................XX.....................] duration: 5.888105424s trace: https://jaeger.dev.storj.io/trace/4173ea2fcd946b89 FAIL (2025-12-30 17:51:06 UTC) orbiter warning: context deadline exceeded Duration: 5.888129911s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [..X.XXXXX.XX.X.XXX......XX.XXX...XX..X.....X.XXX....XX...XX.] duration: 6.059805035s trace: https://jaeger.dev.storj.io/trace/10460225ce957056 FAIL (2025-12-30 17:50:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: 5NLIAKWHQVIA2RMCE2C7KAMKRH5FAUUSZ5DDX3USZWLPUDUJG3TA) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: EPZHSLMNTCIJH6RXDYTTHXY5YMJJ2MMEV7IBZWCU6SHPNKWL6JQA) context deadline exceeded ======LOGS====== segments... Dec 30 17:49:49.437020909 (602h20m49.308103954s) [upload=527944]: Got next segment. Inline: false Dec 30 17:49:49.437067969 (602h20m49.308151044s) [upload=527944]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 17:49:49.437076215 (602h20m49.30815929s) [upload=527944]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 17:49:49.686951542 (602h20m49.558034577s) [seg_pos={0 0}, upload=527944]: Begin upload segment... Dec 30 17:49:49.687004073 (602h20m49.558087118s) [seg_pos={0 0}, upload=527944]: Done begin upload segment. Dec 30 17:49:49.687037827 (602h20m49.558120872s) [noise=true , address=145.79.227.10:4401 , piece_id=QJ3PMF6ZZ3S7CXCHUDYG4TEXQCQ77VFZIPTID3XQBH2W67QGWX4A, seg_pos={0 0}, upload=527944]: Uploading piece... Dec 30 17:49:49.687052695 (602h20m49.55813573s) [noise=true , address=145.79.227.27:4691 , piece_id=GJSMJRZ3O5F4F4UYMB4GBCWU62TEIHJ4QS6EWUD54DTW4HW7I4DQ, seg_pos={0 0}, upload=527944]: Uploading piece... Dec 30 17:49:49.687720466 (602h20m49.558803511s) [noise=true , address=145.79.227.37:4566 , piece_id=OOACMPSVYGRW5JF4LCKL5NIQUMLGCJV5R4E2ADJWPQXARK2G7Q6Q, seg_pos={0 0}, upload=527944]: Uploading piece... Dec 30 17:49:49.688252199 (602h20m49.559335244s) [noise=true , address=145.79.227.11:4729 , piece_id=R77ULVWH7FGT3GHAAZLDSTSKGWEAXFQ2BHZIL3XUYKRF4STP6QOA, seg_pos={0 0}, upload=527944]: Uploading piece... Dec 30 17:49:49.688780645 (602h20m49.55986368s) [noise=true , address=145.79.227.13:4803 , piece_id=ZJXNENZDY4JYPRGMP2UFTBB5DWW3BRI2D257AS26TBVNMTBXYTQQ, seg_pos={0 0}, upload=527944]: Uploading piece... Dec 30 17:49:49.689009481 (602h20m49.560092516s) [noise=true , address=45.91.116.14:4192 , piece_id=5NLIAKWHQVIA2RMCE2C7KAMKRH5FAUUSZ5DDX3USZWLPUDUJG3TA, seg_pos={0 0}, upload=527944]: Uploading piece... Dec 30 17:49:49.689557204 (602h20m49.560640249s) [noise=true , address=45.91.116.8:4046 , piece_id=JDQX53UFVQM4Y653334N65KC4TVZW4Z7Q4BC4SKOUWQLKWNMYRUA, seg_pos={0 0}, upload=527944]: Uploading piece... Dec 30 17:49:49.690094196 (602h20m49.561177231s) [noise=true , address=45.91.116.10:4084 , piece_id=EPZHSLMNTCIJH6RXDYTTHXY5YMJJ2MMEV7IBZWCU6SHPNKWL6JQA, seg_pos={0 0}, upload=527944]: Uploading piece... Dec 30 17:49:49.690136877 (602h20m49.561219912s) [noise=true , address=145.79.227.46:4603 , piece_id=IGYOIZSG264W3PXIABZGXSSGZQ3QINEOFCLRRCZFA5O4L2WLHFIQ, seg_pos={0 0}, upload=527944]: Uploading piece... Dec 30 17:49:49.690521660 (602h20m49.561604695s) [noise=true , address=77.90.31.19:4842 , piece_id=7FJAPWT2U6QWEI5D7XD3XO4SFQFHMHLLUXBV377R6FUADJYCHT5Q, seg_pos={0 0}, upload=527944]: Uploading piece... Dec 30 17:49:49.690573599 (602h20m49.561656644s) [noise=true , address=145.79.227.32:4835 , piece_id=CBR54YF2TU4FFVJZBEBQWZXDNKUNDKTFZZ4456URSZWBLCH6DMDA, seg_pos={0 0}, upload=527944]: Uploading piece... Dec 30 17:49:49.690910671 (602h20m49.561993706s) [noise=true , address=77.90.31.10:4517 , piece_id=J5OUJXEQVRDPRVOKPC6TFR5RLAJOHVU5Q3NRWZI7D3PWEY7TXWTQ, seg_pos={0 0}, upload=527944]: Uploading piece... Dec 30 17:49:49.691025410 (602h20m49.562108455s) [noise=true , address=45.91.116.9:4263 , piece_id=FDL3AM6VKIDSBOLOJLMSRT2VW5OBUCZF67B7PNVYQ6VI7BDRC6QA, seg_pos={0 0}, upload=527944]: Uploading piece... Dec 30 17:49:49.691412136 (602h20m49.562495171s) [noise=true , address=145.79.227.14:4837 , piece_id=KBMO45UWTKDEQ6OM7ZG4OFP4ZKIFIJRGZ2FU56OLBHIF7MOA5PFQ, seg_pos={0 0}, upload=527944]: Uploading piece... Dec 30 17:49:49.691605324 (602h20m49.562688369s) [noise=true , address=145.79.227.50:4374 , piece_id=7RJ7JFON7MECO6DCIP2HAENYTULR2IEABCFBVFAWDJNJJ2HIOVMA, seg_pos={0 0}, upload=527944]: Uploading piece... Dec 30 17:49:49.692056172 (602h20m49.563139217s) [noise=true , address=45.91.116.13:4154 , piece_id=FZRLZHBKKOGN3UXXHOLL7MLDTA77KAPMTPKFBTFLBHQBPHHCSDIA, seg_pos={0 0}, upload=527944]: Uploading piece... Dec 30 17:49:51.384972578 (602h20m51.256055623s) [upload=527944]: Next returned nil segment Dec 30 17:49:51.384990382 (602h20m51.256073427s) [upload=527944]: Waiting for error group managing segments... Dec 30 17:49:51.671969430 (602h20m51.543052465s) [noise=true , address=145.79.227.10:4401 , piece_id=QJ3PMF6ZZ3S7CXCHUDYG4TEXQCQ77VFZIPTID3XQBH2W67QGWX4A, seg_pos={0 0}, upload=527944]: Done uploading piece. err: <nil> Dec 30 17:49:51.674037237 (602h20m51.545120282s) [noise=true , address=145.79.227.11:4729 , piece_id=R77ULVWH7FGT3GHAAZLDSTSKGWEAXFQ2BHZIL3XUYKRF4STP6QOA, seg_pos={0 0}, upload=527944]: Done uploading piece. err: <nil> Dec 30 17:49:51.674211449 (602h20m51.545294484s) [noise=true , address=145.79.227.27:4691 , piece_id=GJSMJRZ3O5F4F4UYMB4GBCWU62TEIHJ4QS6EWUD54DTW4HW7I4DQ, seg_pos={0 0}, upload=527944]: Done uploading piece. err: <nil> Dec 30 17:49:51.679172376 (602h20m51.550255411s) [noise=true , address=145.79.227.14:4837 , piece_id=KBMO45UWTKDEQ6OM7ZG4OFP4ZKIFIJRGZ2FU56OLBHIF7MOA5PFQ, seg_pos={0 0}, upload=527944]: Done uploading piece. err: <nil> Dec 30 17:49:51.695605306 (602h20m51.566688351s) [noise=true , address=145.79.227.50:4374 , piece_id=7RJ7JFON7MECO6DCIP2HAENYTULR2IEABCFBVFAWDJNJJ2HIOVMA, seg_pos={0 0}, upload=527944]: Done uploading piece. err: <nil> Dec 30 17:49:51.703017330 (602h20m51.574100375s) [noise=true , address=145.79.227.37:4566 , piece_id=OOACMPSVYGRW5JF4LCKL5NIQUMLGCJV5R4E2ADJWPQXARK2G7Q6Q, seg_pos={0 0}, upload=527944]: Done uploading piece. err: <nil> Dec 30 17:49:51.711886738 (602h20m51.582969783s) [noise=true , address=145.79.227.13:4803 , piece_id=ZJXNENZDY4JYPRGMP2UFTBB5DWW3BRI2D257AS26TBVNMTBXYTQQ, seg_pos={0 0}, upload=527944]: Done uploading piece. err: <nil> Dec 30 17:49:51.713518385 (602h20m51.58460143s) [noise=true , address=145.79.227.32:4835 , piece_id=CBR54YF2TU4FFVJZBEBQWZXDNKUNDKTFZZ4456URSZWBLCH6DMDA, seg_pos={0 0}, upload=527944]: Done uploading piece. err: <nil> Dec 30 17:49:51.829228743 (602h20m51.700311778s) [noise=true , address=145.79.227.46:4603 , piece_id=IGYOIZSG264W3PXIABZGXSSGZQ3QINEOFCLRRCZFA5O4L2WLHFIQ, seg_pos={0 0}, upload=527944]: Done uploading piece. err: <nil> Dec 30 17:49:51.949995140 (602h20m51.821078185s) [noise=true , address=77.90.31.19:4842 , piece_id=7FJAPWT2U6QWEI5D7XD3XO4SFQFHMHLLUXBV377R6FUADJYCHT5Q, seg_pos={0 0}, upload=527944]: Done uploading piece. err: <nil> Dec 30 17:49:51.999358174 (602h20m51.870441209s) [noise=true , address=77.90.31.10:4517 , piece_id=J5OUJXEQVRDPRVOKPC6TFR5RLAJOHVU5Q3NRWZI7D3PWEY7TXWTQ, seg_pos={0 0}, upload=527944]: Done uploading piece. err: <nil> Dec 30 17:49:52.023869226 (602h20m51.894952261s) [noise=true , address=45.91.116.10:4084 , piece_id=EPZHSLMNTCIJH6RXDYTTHXY5YMJJ2MMEV7IBZWCU6SHPNKWL6JQA, seg_pos={0 0}, upload=527944]: Done uploading piece. err: <nil> Dec 30 17:49:52.032840308 (602h20m51.903923353s) [noise=true , address=45.91.116.14:4192 , piece_id=5NLIAKWHQVIA2RMCE2C7KAMKRH5FAUUSZ5DDX3USZWLPUDUJG3TA, seg_pos={0 0}, upload=527944]: Done uploading piece. err: <nil> Dec 30 17:49:52.032865997 (602h20m51.903949042s) [seg_pos={0 0}, upload=527944]: Segment reached optimal threshold of 13 pieces. Dec 30 17:49:52.033224339 (602h20m51.904307384s) [noise=true , address=45.91.116.9:4263 , piece_id=FDL3AM6VKIDSBOLOJLMSRT2VW5OBUCZF67B7PNVYQ6VI7BDRC6QA, seg_pos={0 0}, upload=527944]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: write tcp 188.166.225.170:32806->45.91.116.9:4263: use of closed network connection; write tcp 188.166.225.170:32806->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32806->45.91.116.9:4263: use of closed network connection Dec 30 17:49:52.033234770 (602h20m51.904317815s) [noise=true , address=45.91.116.9:4263 , piece_id=FDL3AM6VKIDSBOLOJLMSRT2VW5OBUCZF67B7PNVYQ6VI7BDRC6QA, seg_pos={0 0}, upload=527944]: Long tail cancellation due to optimal threshold Dec 30 17:49:52.033282260 (602h20m51.904365295s) [noise=true , address=45.91.116.13:4154 , piece_id=FZRLZHBKKOGN3UXXHOLL7MLDTA77KAPMTPKFBTFLBHQBPHHCSDIA, seg_pos={0 0}, upload=527944]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 30 17:49:52.033298600 (602h20m51.904381635s) [noise=true , address=45.91.116.13:4154 , piece_id=FZRLZHBKKOGN3UXXHOLL7MLDTA77KAPMTPKFBTFLBHQBPHHCSDIA, seg_pos={0 0}, upload=527944]: Long tail cancellation due to optimal threshold Dec 30 17:49:52.033568114 (602h20m51.904651149s) [noise=true , address=45.91.116.8:4046 , piece_id=JDQX53UFVQM4Y653334N65KC4TVZW4Z7Q4BC4SKOUWQLKWNMYRUA, seg_pos={0 0}, upload=527944]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: write tcp 188.166.225.170:35200->45.91.116.8:4046: use of closed network connection; write tcp 188.166.225.170:35200->45.91.116.8:4046: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35200->45.91.116.8:4046: use of closed network connection Dec 30 17:49:52.033575337 (602h20m51.904658372s) [noise=true , address=45.91.116.8:4046 , piece_id=JDQX53UFVQM4Y653334N65KC4TVZW4Z7Q4BC4SKOUWQLKWNMYRUA, seg_pos={0 0}, upload=527944]: Long tail cancellation due to optimal threshold Dec 30 17:49:52.033607619 (602h20m51.904690654s) [seg_pos={0 0}, upload=527944]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 17:49:52.033703492 (602h20m51.904786537s) [upload=527944]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 17:49:52.033709172 (602h20m51.904792217s) [upload=527944]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 17:49:52.336253859 (602h20m52.207336915s) [upload=527944]: Done uploading segments... Dec 30 17:49:52.336284358 (602h20m52.207367413s) [upload=527944]: Upload finished. err: <nil> Duration: 6.059843087s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [...X.XXXXX.XX.X.XXX......XX.XXX...XX..X.....X.XXX....XX...XX] duration: 30.003381518s trace: https://jaeger.dev.storj.io/trace/10460225ce957056 FAIL (2025-12-30 17:50:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs, Piece ID: 5NLIAKWHQVIA2RMCE2C7KAMKRH5FAUUSZ5DDX3USZWLPUDUJG3TA) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: EPZHSLMNTCIJH6RXDYTTHXY5YMJJ2MMEV7IBZWCU6SHPNKWL6JQA) context deadline exceeded ======LOGS====== segments... Dec 30 17:49:49.437020909 (602h20m49.308103954s) [upload=527944]: Got next segment. Inline: false Dec 30 17:49:49.437067969 (602h20m49.308151044s) [upload=527944]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 17:49:49.437076215 (602h20m49.30815929s) [upload=527944]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 17:49:49.686951542 (602h20m49.558034577s) [seg_pos={0 0}, upload=527944]: Begin upload segment... Dec 30 17:49:49.687004073 (602h20m49.558087118s) [seg_pos={0 0}, upload=527944]: Done begin upload segment. Dec 30 17:49:49.687037827 (602h20m49.558120872s) [noise=true , address=145.79.227.10:4401 , piece_id=QJ3PMF6ZZ3S7CXCHUDYG4TEXQCQ77VFZIPTID3XQBH2W67QGWX4A, seg_pos={0 0}, upload=527944]: Uploading piece... Dec 30 17:49:49.687052695 (602h20m49.55813573s) [noise=true , address=145.79.227.27:4691 , piece_id=GJSMJRZ3O5F4F4UYMB4GBCWU62TEIHJ4QS6EWUD54DTW4HW7I4DQ, seg_pos={0 0}, upload=527944]: Uploading piece... Dec 30 17:49:49.687720466 (602h20m49.558803511s) [noise=true , address=145.79.227.37:4566 , piece_id=OOACMPSVYGRW5JF4LCKL5NIQUMLGCJV5R4E2ADJWPQXARK2G7Q6Q, seg_pos={0 0}, upload=527944]: Uploading piece... Dec 30 17:49:49.688252199 (602h20m49.559335244s) [noise=true , address=145.79.227.11:4729 , piece_id=R77ULVWH7FGT3GHAAZLDSTSKGWEAXFQ2BHZIL3XUYKRF4STP6QOA, seg_pos={0 0}, upload=527944]: Uploading piece... Dec 30 17:49:49.688780645 (602h20m49.55986368s) [noise=true , address=145.79.227.13:4803 , piece_id=ZJXNENZDY4JYPRGMP2UFTBB5DWW3BRI2D257AS26TBVNMTBXYTQQ, seg_pos={0 0}, upload=527944]: Uploading piece... Dec 30 17:49:49.689009481 (602h20m49.560092516s) [noise=true , address=45.91.116.14:4192 , piece_id=5NLIAKWHQVIA2RMCE2C7KAMKRH5FAUUSZ5DDX3USZWLPUDUJG3TA, seg_pos={0 0}, upload=527944]: Uploading piece... Dec 30 17:49:49.689557204 (602h20m49.560640249s) [noise=true , address=45.91.116.8:4046 , piece_id=JDQX53UFVQM4Y653334N65KC4TVZW4Z7Q4BC4SKOUWQLKWNMYRUA, seg_pos={0 0}, upload=527944]: Uploading piece... Dec 30 17:49:49.690094196 (602h20m49.561177231s) [noise=true , address=45.91.116.10:4084 , piece_id=EPZHSLMNTCIJH6RXDYTTHXY5YMJJ2MMEV7IBZWCU6SHPNKWL6JQA, seg_pos={0 0}, upload=527944]: Uploading piece... Dec 30 17:49:49.690136877 (602h20m49.561219912s) [noise=true , address=145.79.227.46:4603 , piece_id=IGYOIZSG264W3PXIABZGXSSGZQ3QINEOFCLRRCZFA5O4L2WLHFIQ, seg_pos={0 0}, upload=527944]: Uploading piece... Dec 30 17:49:49.690521660 (602h20m49.561604695s) [noise=true , address=77.90.31.19:4842 , piece_id=7FJAPWT2U6QWEI5D7XD3XO4SFQFHMHLLUXBV377R6FUADJYCHT5Q, seg_pos={0 0}, upload=527944]: Uploading piece... Dec 30 17:49:49.690573599 (602h20m49.561656644s) [noise=true , address=145.79.227.32:4835 , piece_id=CBR54YF2TU4FFVJZBEBQWZXDNKUNDKTFZZ4456URSZWBLCH6DMDA, seg_pos={0 0}, upload=527944]: Uploading piece... Dec 30 17:49:49.690910671 (602h20m49.561993706s) [noise=true , address=77.90.31.10:4517 , piece_id=J5OUJXEQVRDPRVOKPC6TFR5RLAJOHVU5Q3NRWZI7D3PWEY7TXWTQ, seg_pos={0 0}, upload=527944]: Uploading piece... Dec 30 17:49:49.691025410 (602h20m49.562108455s) [noise=true , address=45.91.116.9:4263 , piece_id=FDL3AM6VKIDSBOLOJLMSRT2VW5OBUCZF67B7PNVYQ6VI7BDRC6QA, seg_pos={0 0}, upload=527944]: Uploading piece... Dec 30 17:49:49.691412136 (602h20m49.562495171s) [noise=true , address=145.79.227.14:4837 , piece_id=KBMO45UWTKDEQ6OM7ZG4OFP4ZKIFIJRGZ2FU56OLBHIF7MOA5PFQ, seg_pos={0 0}, upload=527944]: Uploading piece... Dec 30 17:49:49.691605324 (602h20m49.562688369s) [noise=true , address=145.79.227.50:4374 , piece_id=7RJ7JFON7MECO6DCIP2HAENYTULR2IEABCFBVFAWDJNJJ2HIOVMA, seg_pos={0 0}, upload=527944]: Uploading piece... Dec 30 17:49:49.692056172 (602h20m49.563139217s) [noise=true , address=45.91.116.13:4154 , piece_id=FZRLZHBKKOGN3UXXHOLL7MLDTA77KAPMTPKFBTFLBHQBPHHCSDIA, seg_pos={0 0}, upload=527944]: Uploading piece... Dec 30 17:49:51.384972578 (602h20m51.256055623s) [upload=527944]: Next returned nil segment Dec 30 17:49:51.384990382 (602h20m51.256073427s) [upload=527944]: Waiting for error group managing segments... Dec 30 17:49:51.671969430 (602h20m51.543052465s) [noise=true , address=145.79.227.10:4401 , piece_id=QJ3PMF6ZZ3S7CXCHUDYG4TEXQCQ77VFZIPTID3XQBH2W67QGWX4A, seg_pos={0 0}, upload=527944]: Done uploading piece. err: <nil> Dec 30 17:49:51.674037237 (602h20m51.545120282s) [noise=true , address=145.79.227.11:4729 , piece_id=R77ULVWH7FGT3GHAAZLDSTSKGWEAXFQ2BHZIL3XUYKRF4STP6QOA, seg_pos={0 0}, upload=527944]: Done uploading piece. err: <nil> Dec 30 17:49:51.674211449 (602h20m51.545294484s) [noise=true , address=145.79.227.27:4691 , piece_id=GJSMJRZ3O5F4F4UYMB4GBCWU62TEIHJ4QS6EWUD54DTW4HW7I4DQ, seg_pos={0 0}, upload=527944]: Done uploading piece. err: <nil> Dec 30 17:49:51.679172376 (602h20m51.550255411s) [noise=true , address=145.79.227.14:4837 , piece_id=KBMO45UWTKDEQ6OM7ZG4OFP4ZKIFIJRGZ2FU56OLBHIF7MOA5PFQ, seg_pos={0 0}, upload=527944]: Done uploading piece. err: <nil> Dec 30 17:49:51.695605306 (602h20m51.566688351s) [noise=true , address=145.79.227.50:4374 , piece_id=7RJ7JFON7MECO6DCIP2HAENYTULR2IEABCFBVFAWDJNJJ2HIOVMA, seg_pos={0 0}, upload=527944]: Done uploading piece. err: <nil> Dec 30 17:49:51.703017330 (602h20m51.574100375s) [noise=true , address=145.79.227.37:4566 , piece_id=OOACMPSVYGRW5JF4LCKL5NIQUMLGCJV5R4E2ADJWPQXARK2G7Q6Q, seg_pos={0 0}, upload=527944]: Done uploading piece. err: <nil> Dec 30 17:49:51.711886738 (602h20m51.582969783s) [noise=true , address=145.79.227.13:4803 , piece_id=ZJXNENZDY4JYPRGMP2UFTBB5DWW3BRI2D257AS26TBVNMTBXYTQQ, seg_pos={0 0}, upload=527944]: Done uploading piece. err: <nil> Dec 30 17:49:51.713518385 (602h20m51.58460143s) [noise=true , address=145.79.227.32:4835 , piece_id=CBR54YF2TU4FFVJZBEBQWZXDNKUNDKTFZZ4456URSZWBLCH6DMDA, seg_pos={0 0}, upload=527944]: Done uploading piece. err: <nil> Dec 30 17:49:51.829228743 (602h20m51.700311778s) [noise=true , address=145.79.227.46:4603 , piece_id=IGYOIZSG264W3PXIABZGXSSGZQ3QINEOFCLRRCZFA5O4L2WLHFIQ, seg_pos={0 0}, upload=527944]: Done uploading piece. err: <nil> Dec 30 17:49:51.949995140 (602h20m51.821078185s) [noise=true , address=77.90.31.19:4842 , piece_id=7FJAPWT2U6QWEI5D7XD3XO4SFQFHMHLLUXBV377R6FUADJYCHT5Q, seg_pos={0 0}, upload=527944]: Done uploading piece. err: <nil> Dec 30 17:49:51.999358174 (602h20m51.870441209s) [noise=true , address=77.90.31.10:4517 , piece_id=J5OUJXEQVRDPRVOKPC6TFR5RLAJOHVU5Q3NRWZI7D3PWEY7TXWTQ, seg_pos={0 0}, upload=527944]: Done uploading piece. err: <nil> Dec 30 17:49:52.023869226 (602h20m51.894952261s) [noise=true , address=45.91.116.10:4084 , piece_id=EPZHSLMNTCIJH6RXDYTTHXY5YMJJ2MMEV7IBZWCU6SHPNKWL6JQA, seg_pos={0 0}, upload=527944]: Done uploading piece. err: <nil> Dec 30 17:49:52.032840308 (602h20m51.903923353s) [noise=true , address=45.91.116.14:4192 , piece_id=5NLIAKWHQVIA2RMCE2C7KAMKRH5FAUUSZ5DDX3USZWLPUDUJG3TA, seg_pos={0 0}, upload=527944]: Done uploading piece. err: <nil> Dec 30 17:49:52.032865997 (602h20m51.903949042s) [seg_pos={0 0}, upload=527944]: Segment reached optimal threshold of 13 pieces. Dec 30 17:49:52.033224339 (602h20m51.904307384s) [noise=true , address=45.91.116.9:4263 , piece_id=FDL3AM6VKIDSBOLOJLMSRT2VW5OBUCZF67B7PNVYQ6VI7BDRC6QA, seg_pos={0 0}, upload=527944]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: write tcp 188.166.225.170:32806->45.91.116.9:4263: use of closed network connection; write tcp 188.166.225.170:32806->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:32806->45.91.116.9:4263: use of closed network connection Dec 30 17:49:52.033234770 (602h20m51.904317815s) [noise=true , address=45.91.116.9:4263 , piece_id=FDL3AM6VKIDSBOLOJLMSRT2VW5OBUCZF67B7PNVYQ6VI7BDRC6QA, seg_pos={0 0}, upload=527944]: Long tail cancellation due to optimal threshold Dec 30 17:49:52.033282260 (602h20m51.904365295s) [noise=true , address=45.91.116.13:4154 , piece_id=FZRLZHBKKOGN3UXXHOLL7MLDTA77KAPMTPKFBTFLBHQBPHHCSDIA, seg_pos={0 0}, upload=527944]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF Dec 30 17:49:52.033298600 (602h20m51.904381635s) [noise=true , address=45.91.116.13:4154 , piece_id=FZRLZHBKKOGN3UXXHOLL7MLDTA77KAPMTPKFBTFLBHQBPHHCSDIA, seg_pos={0 0}, upload=527944]: Long tail cancellation due to optimal threshold Dec 30 17:49:52.033568114 (602h20m51.904651149s) [noise=true , address=45.91.116.8:4046 , piece_id=JDQX53UFVQM4Y653334N65KC4TVZW4Z7Q4BC4SKOUWQLKWNMYRUA, seg_pos={0 0}, upload=527944]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: write tcp 188.166.225.170:35200->45.91.116.8:4046: use of closed network connection; write tcp 188.166.225.170:35200->45.91.116.8:4046: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35200->45.91.116.8:4046: use of closed network connection Dec 30 17:49:52.033575337 (602h20m51.904658372s) [noise=true , address=45.91.116.8:4046 , piece_id=JDQX53UFVQM4Y653334N65KC4TVZW4Z7Q4BC4SKOUWQLKWNMYRUA, seg_pos={0 0}, upload=527944]: Long tail cancellation due to optimal threshold Dec 30 17:49:52.033607619 (602h20m51.904690654s) [seg_pos={0 0}, upload=527944]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 17:49:52.033703492 (602h20m51.904786537s) [upload=527944]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 17:49:52.033709172 (602h20m51.904792217s) [upload=527944]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 17:49:52.336253859 (602h20m52.207336915s) [upload=527944]: Done uploading segments... Dec 30 17:49:52.336284358 (602h20m52.207367413s) [upload=527944]: Upload finished. err: <nil> Duration: 30.003406196s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 23.310211566s trace: https://jaeger.dev.storj.io/trace/5ea8d7d4f61e13a2 FAIL (2025-12-30 17:50:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1X5A7bLqMEQuaMbf9xZ7PDUD4161sz2QqdM9r59y3bsyPBJhaK, Piece ID: HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ) context deadline exceeded; piecestore: (Node ID: 12eLKUg2vTnZMrLaraSPEoGXe3jDVBJYyURdcn4f2gCN5pvvCQ8, Piece ID: KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ) context deadline exceeded; piecestore: (Node ID: 12W4n6uf6azQk1QgeR8Xw7UkpnZm19zdyFKwojCsqZt2GFHrpce, Piece ID: DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ) context deadline exceeded ======LOGS====== 17.114181822 (601h42m16.985264867s) [noise=true , address=145.79.227.44:4208 , piece_id=3OXDCHZSOIFS5VQCI4SXSAQFICDCYHRE5NGFBERR33KOTGSZWLDQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.114641387 (601h42m16.985724432s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.115652532 (601h42m16.986735578s) [noise=true , address=77.90.31.17:4790 , piece_id=5RSRESKGDA2YF4NBRF4ZFABBKWJK4NO6VLEXW2Z6EY3OPXH2WPNA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.115937134 (601h42m16.987020179s) [noise=true , address=64.49.13.155:4923 , piece_id=MUJD7XLM2YRLRCHYVSDPBE6VBYLHCOA7HJQP22ZSET4N574XQMJA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.116814986 (601h42m16.987898031s) [noise=true , address=45.91.116.11:4139 , piece_id=72DPEWZ7ZPZZ53UGYV2VCUKLCK7HCM22CDBRHWVLJGDGO5B5JADQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.117501182 (601h42m16.988584228s) [noise=true , address=77.90.31.14:4665 , piece_id=DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.118263334 (601h42m16.989346379s) [noise=true , address=145.79.227.40:4084 , piece_id=ZADZDRKBIILLAI4IT335CYETOSFOFUUPQIH6HZJCTHKRTZSPDJUQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.118555850 (601h42m16.989638886s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.118765930 (601h42m16.989849036s) [noise=true , address=77.90.31.21:4905 , piece_id=HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.119029533 (601h42m16.990112568s) [noise=true , address=77.90.31.10:4533 , piece_id=QFBBYNQHQL2G7PEREK43FZBRUMKMVNBC2XKTE2NWTVTKN2452LPA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.119417672 (601h42m16.990500707s) [noise=true , address=45.91.116.12:4258 , piece_id=KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.119441588 (601h42m16.990524643s) [noise=true , address=145.79.227.40:4053 , piece_id=JCSIUMWKEAQNQXTNSJTN6VSB3V2SIDRQK67QMK46B4GOUWEMQE2A, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.119763470 (601h42m16.990846505s) [noise=true , address=77.90.31.18:4821 , piece_id=MXZFKXM3JV2HQURDLIU7BKWLAKDZ27O4ER3SGC7QBZV5I3J2EHKA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.120028685 (601h42m16.99111172s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.120086986 (601h42m16.991170031s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.120586507 (601h42m16.991669562s) [noise=true , address=45.91.116.13:4168 , piece_id=AKUE5H3A2UECQOZY3GV3MWZKIHSNKCDOG32DKCNHIRTJROJ74M3A, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.121258768 (601h42m16.992341813s) [noise=true , address=145.79.227.9:4379 , piece_id=QTFXFSGGKYBQSPWJYBBJMQUL3D5EEO3IX4C53ECD7FXMUOGUBC7Q, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:18.568739165 (601h42m18.43982221s) [upload=527370]: Next returned nil segment Dec 30 17:11:18.568751328 (601h42m18.439834363s) [upload=527370]: Waiting for error group managing segments... Dec 30 17:11:18.978424178 (601h42m18.849507213s) [noise=true , address=64.49.13.155:4923 , piece_id=MUJD7XLM2YRLRCHYVSDPBE6VBYLHCOA7HJQP22ZSET4N574XQMJA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:18.986258125 (601h42m18.85734116s) [noise=true , address=145.79.227.9:4379 , piece_id=QTFXFSGGKYBQSPWJYBBJMQUL3D5EEO3IX4C53ECD7FXMUOGUBC7Q, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.274519421 (601h42m19.145602456s) [noise=true , address=77.90.31.14:4665 , piece_id=DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.315775438 (601h42m19.186858473s) [noise=true , address=45.91.116.13:4168 , piece_id=AKUE5H3A2UECQOZY3GV3MWZKIHSNKCDOG32DKCNHIRTJROJ74M3A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.335765371 (601h42m19.206848406s) [noise=true , address=77.90.31.21:4905 , piece_id=HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.354812129 (601h42m19.225895164s) [noise=true , address=77.90.31.10:4533 , piece_id=QFBBYNQHQL2G7PEREK43FZBRUMKMVNBC2XKTE2NWTVTKN2452LPA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.392024917 (601h42m19.263107952s) [noise=true , address=45.91.116.11:4139 , piece_id=72DPEWZ7ZPZZ53UGYV2VCUKLCK7HCM22CDBRHWVLJGDGO5B5JADQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.396084327 (601h42m19.267167372s) [noise=true , address=77.90.31.18:4821 , piece_id=MXZFKXM3JV2HQURDLIU7BKWLAKDZ27O4ER3SGC7QBZV5I3J2EHKA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.862099099 (601h42m19.733182134s) [noise=true , address=77.90.31.17:4790 , piece_id=5RSRESKGDA2YF4NBRF4ZFABBKWJK4NO6VLEXW2Z6EY3OPXH2WPNA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.905412913 (601h42m19.776495948s) [noise=true , address=145.79.227.44:4208 , piece_id=3OXDCHZSOIFS5VQCI4SXSAQFICDCYHRE5NGFBERR33KOTGSZWLDQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.910309688 (601h42m19.781392743s) [noise=true , address=145.79.227.30:4794 , piece_id=EJH6Q3T2BMPVLG6F7CW7N4GYOA3EZFPADL6VPT7WKKG65B2WZVXQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.934233361 (601h42m19.805316397s) [noise=true , address=145.79.227.40:4084 , piece_id=ZADZDRKBIILLAI4IT335CYETOSFOFUUPQIH6HZJCTHKRTZSPDJUQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:20.074011527 (601h42m19.945094572s) [noise=true , address=45.91.116.12:4258 , piece_id=KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:20.106010337 (601h42m19.977093372s) [noise=true , address=145.79.227.40:4053 , piece_id=JCSIUMWKEAQNQXTNSJTN6VSB3V2SIDRQK67QMK46B4GOUWEMQE2A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:20.106080911 (601h42m19.977163946s) [seg_pos={0 0}, upload=527370]: Segment reached optimal threshold of 14 pieces. Dec 30 17:11:20.106274520 (601h42m19.977357565s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Dec 30 17:11:20.106281644 (601h42m19.977364679s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold Dec 30 17:11:20.106351607 (601h42m19.977434642s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF Dec 30 17:11:20.106367767 (601h42m19.977450802s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold Dec 30 17:11:20.106424967 (601h42m19.977508002s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Dec 30 17:11:20.106432451 (601h42m19.977515486s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold Dec 30 17:11:20.106540666 (601h42m19.977623702s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 30 17:11:20.106546207 (601h42m19.977629242s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold Dec 30 17:11:20.106563380 (601h42m19.977646415s) [seg_pos={0 0}, upload=527370]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 17:11:20.106625529 (601h42m19.977708564s) [upload=527370]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 17:11:20.106629837 (601h42m19.977712882s) [upload=527370]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 17:11:20.406465452 (601h42m20.277548507s) [upload=527370]: Done uploading segments... Dec 30 17:11:20.406504206 (601h42m20.277587251s) [upload=527370]: Upload finished. err: <nil> Duration: 23.31023511s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [..X....X.XX..X..X......X.........X..XXX...................X.] duration: 3.630689432s trace: https://jaeger.dev.storj.io/trace/14885d465b83b494 FAIL (2025-12-30 17:50:13 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.630791114s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X...X..X............................XX....................] duration: 6.177397484s trace: https://jaeger.dev.storj.io/trace/4173ea2fcd946b89 FAIL (2025-12-30 17:50:11 UTC) orbiter warning: context deadline exceeded Duration: 6.177429264s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X...X..X............................XX...................] duration: 6.106016062s trace: https://jaeger.dev.storj.io/trace/4173ea2fcd946b89 FAIL (2025-12-30 17:49:37 UTC) orbiter warning: context deadline exceeded Duration: 6.106044877s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [....X.XXXXX.XX.X.XXX......XX.XXX...XX..X.....X.XXX....XX...X] duration: 30.004432386s trace: https://jaeger.dev.storj.io/trace/e307178d1adf8c5 FAIL (2025-12-30 17:49:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV, Piece ID: F7QTHLQ5DVER5XRG3WLN6JTIN75CYVQSTDF3V4LHDFZD3ICIOPIA) context deadline exceeded; piecestore: (Node ID: 12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, Piece ID: Y7ERTKCVTNRWCMKFBJMQBE6YBR3G4DLQQWRQIDNGGTHNEJMRCSYQ) context deadline exceeded ======LOGS====== Dec 30 17:49:06.279599274 (602h20m6.150682309s) [upload=527934]: Starting upload Dec 30 17:49:06.279677443 (602h20m6.150760488s) [upload=527934]: Done starting upload Dec 30 17:49:06.279753968 (602h20m6.150837003s) [upload=527934]: Uploading segments... Dec 30 17:49:06.279883715 (602h20m6.150966761s) [upload=527934]: Got next segment. Inline: false Dec 30 17:49:06.279923521 (602h20m6.151006556s) [upload=527934]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 17:49:06.279929152 (602h20m6.151012207s) [upload=527934]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 17:49:06.526823723 (602h20m6.397906758s) [seg_pos={0 0}, upload=527934]: Begin upload segment... Dec 30 17:49:06.526866275 (602h20m6.39794931s) [seg_pos={0 0}, upload=527934]: Done begin upload segment. Dec 30 17:49:06.526894638 (602h20m6.397977683s) [noise=true , address=145.79.227.38:4977 , piece_id=P3QSXNWIM6NLGCR2ZDEUHDGNHGEWFVEZGEN2PZ7IHSQ7FR42XSLA, seg_pos={0 0}, upload=527934]: Uploading piece... Dec 30 17:49:06.526946407 (602h20m6.398029452s) [noise=true , address=77.90.31.10:4518 , piece_id=KXQQVXUQ2KGAJH55CR32CUYIUY4V7XDL6LLJIQAMIOCW23ETPS5Q, seg_pos={0 0}, upload=527934]: Uploading piece... Dec 30 17:49:06.528875741 (602h20m6.399958786s) [noise=true , address=145.79.227.47:4267 , piece_id=Y4ZHQ4D5S4H6YYNKBIOV3V6AAAQNYML6NTFJFQ24WWJLDPOH27VQ, seg_pos={0 0}, upload=527934]: Uploading piece... Dec 30 17:49:06.529311711 (602h20m6.400394746s) [noise=true , address=145.79.227.45:4231 , piece_id=4VLNHQIRU52J22VN4YYYLSJFXUZOF23NOMYU5Q5LMB3NMILLUXMQ, seg_pos={0 0}, upload=527934]: Uploading piece... Dec 30 17:49:06.529725639 (602h20m6.400808675s) [noise=true , address=45.91.116.12:4226 , piece_id=Y7ERTKCVTNRWCMKFBJMQBE6YBR3G4DLQQWRQIDNGGTHNEJMRCSYQ, seg_pos={0 0}, upload=527934]: Uploading piece... Dec 30 17:49:06.530874026 (602h20m6.401957061s) [noise=true , address=145.79.227.3:4151 , piece_id=7R3AOXDR4HZFHROYEYBCWULIRFTOQKPDNZP3Q3KLLSPRUT3U6QCQ, seg_pos={0 0}, upload=527934]: Uploading piece... Dec 30 17:49:06.532251849 (602h20m6.403334884s) [noise=true , address=145.79.227.6:4257 , piece_id=3MSS5JU2XOZOVFKLL6SFTKEUD4ELCWZ72ZKZUQTPZ7KW6CQE776A, seg_pos={0 0}, upload=527934]: Uploading piece... Dec 30 17:49:06.533370990 (602h20m6.404454025s) [noise=true , address=45.91.116.5:4300 , piece_id=CYPFRWQXOOMGLEZWGNUJ6TTMZESMPPBYCUHZ5QWKVHIPECI4N6EA, seg_pos={0 0}, upload=527934]: Uploading piece... Dec 30 17:49:06.534708658 (602h20m6.405791693s) [noise=true , address=145.79.227.52:4447 , piece_id=GAETR7FRJYHZ5QUAA3AXH3UTJHESO4RIKGPJLTXYSQBQPYZYUTSQ, seg_pos={0 0}, upload=527934]: Uploading piece... Dec 30 17:49:06.535745853 (602h20m6.406828888s) [noise=true , address=77.90.31.17:4770 , piece_id=PPRSHLNL224WTQT742J5NXZQQDVBEW4XZMSR4LVJX4EO6ILEWXTQ, seg_pos={0 0}, upload=527934]: Uploading piece... Dec 30 17:49:06.537008497 (602h20m6.408091532s) [noise=true , address=45.91.116.14:4189 , piece_id=EGBEQPJ2FRH5FV55JQUUW52DKSBMKW6GD5MQYALKVORKHBDZVYIQ, seg_pos={0 0}, upload=527934]: Uploading piece... Dec 30 17:49:06.537080915 (602h20m6.40816395s) [noise=true , address=45.91.116.9:4261 , piece_id=4SIKU6JTCSFFSVIJ46KJSOOD7L3GAGHTS7MR4435BXJTZ6JH5FDQ, seg_pos={0 0}, upload=527934]: Uploading piece... Dec 30 17:49:06.538659080 (602h20m6.409742115s) [noise=true , address=77.90.31.21:4880 , piece_id=XYN5ILKRSRJEB2HEIW4NVXDRATOKGZ7ERFZ5VGDRO3V4IJQDXQFQ, seg_pos={0 0}, upload=527934]: Uploading piece... Dec 30 17:49:06.539763844 (602h20m6.410846879s) [noise=true , address=45.91.116.6:4973 , piece_id=F7QTHLQ5DVER5XRG3WLN6JTIN75CYVQSTDF3V4LHDFZD3ICIOPIA, seg_pos={0 0}, upload=527934]: Uploading piece... Dec 30 17:49:06.540653989 (602h20m6.411737034s) [noise=true , address=77.90.31.15:4697 , piece_id=H7ZEDJODAOVB627BDNOXUYULTO3MDQJAYJIAQITSWVHGCGZ3SGXA, seg_pos={0 0}, upload=527934]: Uploading piece... Dec 30 17:49:06.541551607 (602h20m6.412634653s) [noise=true , address=145.79.227.48:4301 , piece_id=AUC67R5SK3VUCMSXNMF4KE4Q3OAFI2GML2N7JP24T53AHUOHOPPQ, seg_pos={0 0}, upload=527934]: Uploading piece... Dec 30 17:49:08.226342714 (602h20m8.097425759s) [upload=527934]: Next returned nil segment Dec 30 17:49:08.226355618 (602h20m8.097438663s) [upload=527934]: Waiting for error group managing segments... Dec 30 17:49:08.518190312 (602h20m8.389273358s) [noise=true , address=145.79.227.52:4447 , piece_id=GAETR7FRJYHZ5QUAA3AXH3UTJHESO4RIKGPJLTXYSQBQPYZYUTSQ, seg_pos={0 0}, upload=527934]: Done uploading piece. err: <nil> Dec 30 17:49:08.520413024 (602h20m8.391496069s) [noise=true , address=145.79.227.6:4257 , piece_id=3MSS5JU2XOZOVFKLL6SFTKEUD4ELCWZ72ZKZUQTPZ7KW6CQE776A, seg_pos={0 0}, upload=527934]: Done uploading piece. err: <nil> Dec 30 17:49:08.520460335 (602h20m8.39154337s) [noise=true , address=145.79.227.48:4301 , piece_id=AUC67R5SK3VUCMSXNMF4KE4Q3OAFI2GML2N7JP24T53AHUOHOPPQ, seg_pos={0 0}, upload=527934]: Done uploading piece. err: <nil> Dec 30 17:49:08.525697196 (602h20m8.396780241s) [noise=true , address=145.79.227.47:4267 , piece_id=Y4ZHQ4D5S4H6YYNKBIOV3V6AAAQNYML6NTFJFQ24WWJLDPOH27VQ, seg_pos={0 0}, upload=527934]: Done uploading piece. err: <nil> Dec 30 17:49:08.538806438 (602h20m8.409889483s) [noise=true , address=145.79.227.38:4977 , piece_id=P3QSXNWIM6NLGCR2ZDEUHDGNHGEWFVEZGEN2PZ7IHSQ7FR42XSLA, seg_pos={0 0}, upload=527934]: Done uploading piece. err: <nil> Dec 30 17:49:08.555375298 (602h20m8.426458343s) [noise=true , address=145.79.227.45:4231 , piece_id=4VLNHQIRU52J22VN4YYYLSJFXUZOF23NOMYU5Q5LMB3NMILLUXMQ, seg_pos={0 0}, upload=527934]: Done uploading piece. err: <nil> Dec 30 17:49:08.797170719 (602h20m8.668253754s) [noise=true , address=77.90.31.17:4770 , piece_id=PPRSHLNL224WTQT742J5NXZQQDVBEW4XZMSR4LVJX4EO6ILEWXTQ, seg_pos={0 0}, upload=527934]: Done uploading piece. err: <nil> Dec 30 17:49:08.814857428 (602h20m8.685940473s) [noise=true , address=45.91.116.12:4226 , piece_id=Y7ERTKCVTNRWCMKFBJMQBE6YBR3G4DLQQWRQIDNGGTHNEJMRCSYQ, seg_pos={0 0}, upload=527934]: Done uploading piece. err: <nil> Dec 30 17:49:08.859948788 (602h20m8.731031823s) [noise=true , address=45.91.116.6:4973 , piece_id=F7QTHLQ5DVER5XRG3WLN6JTIN75CYVQSTDF3V4LHDFZD3ICIOPIA, seg_pos={0 0}, upload=527934]: Done uploading piece. err: <nil> Dec 30 17:49:08.911914137 (602h20m8.782997182s) [noise=true , address=77.90.31.15:4697 , piece_id=H7ZEDJODAOVB627BDNOXUYULTO3MDQJAYJIAQITSWVHGCGZ3SGXA, seg_pos={0 0}, upload=527934]: Done uploading piece. err: <nil> Dec 30 17:49:09.475130770 (602h20m9.346213815s) [noise=true , address=77.90.31.21:4880 , piece_id=XYN5ILKRSRJEB2HEIW4NVXDRATOKGZ7ERFZ5VGDRO3V4IJQDXQFQ, seg_pos={0 0}, upload=527934]: Done uploading piece. err: <nil> Dec 30 17:49:09.494806535 (602h20m9.36588957s) [noise=true , address=77.90.31.10:4518 , piece_id=KXQQVXUQ2KGAJH55CR32CUYIUY4V7XDL6LLJIQAMIOCW23ETPS5Q, seg_pos={0 0}, upload=527934]: Done uploading piece. err: <nil> Dec 30 17:49:09.580795730 (602h20m9.451878776s) [noise=true , address=45.91.116.9:4261 , piece_id=4SIKU6JTCSFFSVIJ46KJSOOD7L3GAGHTS7MR4435BXJTZ6JH5FDQ, seg_pos={0 0}, upload=527934]: Done uploading piece. err: <nil> Dec 30 17:49:09.580829445 (602h20m9.45191248s) [seg_pos={0 0}, upload=527934]: Segment reached optimal threshold of 13 pieces. Dec 30 17:49:09.581133653 (602h20m9.452216688s) [noise=true , address=45.91.116.14:4189 , piece_id=EGBEQPJ2FRH5FV55JQUUW52DKSBMKW6GD5MQYALKVORKHBDZVYIQ, seg_pos={0 0}, upload=527934]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 30 17:49:09.581142681 (602h20m9.452225726s) [noise=true , address=45.91.116.14:4189 , piece_id=EGBEQPJ2FRH5FV55JQUUW52DKSBMKW6GD5MQYALKVORKHBDZVYIQ, seg_pos={0 0}, upload=527934]: Long tail cancellation due to optimal threshold Dec 30 17:49:09.581157058 (602h20m9.452240093s) [noise=true , address=45.91.116.5:4300 , piece_id=CYPFRWQXOOMGLEZWGNUJ6TTMZESMPPBYCUHZ5QWKVHIPECI4N6EA, seg_pos={0 0}, upload=527934]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 30 17:49:09.581175784 (602h20m9.452258819s) [noise=true , address=45.91.116.5:4300 , piece_id=CYPFRWQXOOMGLEZWGNUJ6TTMZESMPPBYCUHZ5QWKVHIPECI4N6EA, seg_pos={0 0}, upload=527934]: Long tail cancellation due to optimal threshold Dec 30 17:49:09.581388639 (602h20m9.452471674s) [noise=true , address=145.79.227.3:4151 , piece_id=7R3AOXDR4HZFHROYEYBCWULIRFTOQKPDNZP3Q3KLLSPRUT3U6QCQ, seg_pos={0 0}, upload=527934]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:36842->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:36842->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36842->145.79.227.3:4151: use of closed network connection Dec 30 17:49:09.581395683 (602h20m9.452478718s) [noise=true , address=145.79.227.3:4151 , piece_id=7R3AOXDR4HZFHROYEYBCWULIRFTOQKPDNZP3Q3KLLSPRUT3U6QCQ, seg_pos={0 0}, upload=527934]: Long tail cancellation due to optimal threshold Dec 30 17:49:09.581435328 (602h20m9.452518373s) [seg_pos={0 0}, upload=527934]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 17:49:09.581541660 (602h20m9.452624705s) [upload=527934]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 17:49:09.581549545 (602h20m9.45263258s) [upload=527934]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 17:49:09.877237616 (602h20m9.748320651s) [upload=527934]: Done uploading segments... Dec 30 17:49:09.877255700 (602h20m9.748338735s) [upload=527934]: Upload finished. err: <nil> Duration: 30.00447148s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 13 out of 60 is failed (threshold=6) [X..X....X.XX..X..X......X.........X..XXX...................X] duration: 5.000989846s trace: https://jaeger.dev.storj.io/trace/14885d465b83b494 FAIL (2025-12-30 17:49:31 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001011178s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 33.893747166s trace: https://jaeger.dev.storj.io/trace/5ea8d7d4f61e13a2 FAIL (2025-12-30 17:49:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1X5A7bLqMEQuaMbf9xZ7PDUD4161sz2QqdM9r59y3bsyPBJhaK, Piece ID: HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ) context deadline exceeded; piecestore: (Node ID: 12eLKUg2vTnZMrLaraSPEoGXe3jDVBJYyURdcn4f2gCN5pvvCQ8, Piece ID: KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ) context deadline exceeded; piecestore: (Node ID: 12W4n6uf6azQk1QgeR8Xw7UkpnZm19zdyFKwojCsqZt2GFHrpce, Piece ID: DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ) context deadline exceeded ======LOGS====== 17.114181822 (601h42m16.985264867s) [noise=true , address=145.79.227.44:4208 , piece_id=3OXDCHZSOIFS5VQCI4SXSAQFICDCYHRE5NGFBERR33KOTGSZWLDQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.114641387 (601h42m16.985724432s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.115652532 (601h42m16.986735578s) [noise=true , address=77.90.31.17:4790 , piece_id=5RSRESKGDA2YF4NBRF4ZFABBKWJK4NO6VLEXW2Z6EY3OPXH2WPNA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.115937134 (601h42m16.987020179s) [noise=true , address=64.49.13.155:4923 , piece_id=MUJD7XLM2YRLRCHYVSDPBE6VBYLHCOA7HJQP22ZSET4N574XQMJA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.116814986 (601h42m16.987898031s) [noise=true , address=45.91.116.11:4139 , piece_id=72DPEWZ7ZPZZ53UGYV2VCUKLCK7HCM22CDBRHWVLJGDGO5B5JADQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.117501182 (601h42m16.988584228s) [noise=true , address=77.90.31.14:4665 , piece_id=DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.118263334 (601h42m16.989346379s) [noise=true , address=145.79.227.40:4084 , piece_id=ZADZDRKBIILLAI4IT335CYETOSFOFUUPQIH6HZJCTHKRTZSPDJUQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.118555850 (601h42m16.989638886s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.118765930 (601h42m16.989849036s) [noise=true , address=77.90.31.21:4905 , piece_id=HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.119029533 (601h42m16.990112568s) [noise=true , address=77.90.31.10:4533 , piece_id=QFBBYNQHQL2G7PEREK43FZBRUMKMVNBC2XKTE2NWTVTKN2452LPA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.119417672 (601h42m16.990500707s) [noise=true , address=45.91.116.12:4258 , piece_id=KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.119441588 (601h42m16.990524643s) [noise=true , address=145.79.227.40:4053 , piece_id=JCSIUMWKEAQNQXTNSJTN6VSB3V2SIDRQK67QMK46B4GOUWEMQE2A, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.119763470 (601h42m16.990846505s) [noise=true , address=77.90.31.18:4821 , piece_id=MXZFKXM3JV2HQURDLIU7BKWLAKDZ27O4ER3SGC7QBZV5I3J2EHKA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.120028685 (601h42m16.99111172s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.120086986 (601h42m16.991170031s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.120586507 (601h42m16.991669562s) [noise=true , address=45.91.116.13:4168 , piece_id=AKUE5H3A2UECQOZY3GV3MWZKIHSNKCDOG32DKCNHIRTJROJ74M3A, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.121258768 (601h42m16.992341813s) [noise=true , address=145.79.227.9:4379 , piece_id=QTFXFSGGKYBQSPWJYBBJMQUL3D5EEO3IX4C53ECD7FXMUOGUBC7Q, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:18.568739165 (601h42m18.43982221s) [upload=527370]: Next returned nil segment Dec 30 17:11:18.568751328 (601h42m18.439834363s) [upload=527370]: Waiting for error group managing segments... Dec 30 17:11:18.978424178 (601h42m18.849507213s) [noise=true , address=64.49.13.155:4923 , piece_id=MUJD7XLM2YRLRCHYVSDPBE6VBYLHCOA7HJQP22ZSET4N574XQMJA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:18.986258125 (601h42m18.85734116s) [noise=true , address=145.79.227.9:4379 , piece_id=QTFXFSGGKYBQSPWJYBBJMQUL3D5EEO3IX4C53ECD7FXMUOGUBC7Q, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.274519421 (601h42m19.145602456s) [noise=true , address=77.90.31.14:4665 , piece_id=DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.315775438 (601h42m19.186858473s) [noise=true , address=45.91.116.13:4168 , piece_id=AKUE5H3A2UECQOZY3GV3MWZKIHSNKCDOG32DKCNHIRTJROJ74M3A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.335765371 (601h42m19.206848406s) [noise=true , address=77.90.31.21:4905 , piece_id=HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.354812129 (601h42m19.225895164s) [noise=true , address=77.90.31.10:4533 , piece_id=QFBBYNQHQL2G7PEREK43FZBRUMKMVNBC2XKTE2NWTVTKN2452LPA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.392024917 (601h42m19.263107952s) [noise=true , address=45.91.116.11:4139 , piece_id=72DPEWZ7ZPZZ53UGYV2VCUKLCK7HCM22CDBRHWVLJGDGO5B5JADQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.396084327 (601h42m19.267167372s) [noise=true , address=77.90.31.18:4821 , piece_id=MXZFKXM3JV2HQURDLIU7BKWLAKDZ27O4ER3SGC7QBZV5I3J2EHKA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.862099099 (601h42m19.733182134s) [noise=true , address=77.90.31.17:4790 , piece_id=5RSRESKGDA2YF4NBRF4ZFABBKWJK4NO6VLEXW2Z6EY3OPXH2WPNA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.905412913 (601h42m19.776495948s) [noise=true , address=145.79.227.44:4208 , piece_id=3OXDCHZSOIFS5VQCI4SXSAQFICDCYHRE5NGFBERR33KOTGSZWLDQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.910309688 (601h42m19.781392743s) [noise=true , address=145.79.227.30:4794 , piece_id=EJH6Q3T2BMPVLG6F7CW7N4GYOA3EZFPADL6VPT7WKKG65B2WZVXQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.934233361 (601h42m19.805316397s) [noise=true , address=145.79.227.40:4084 , piece_id=ZADZDRKBIILLAI4IT335CYETOSFOFUUPQIH6HZJCTHKRTZSPDJUQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:20.074011527 (601h42m19.945094572s) [noise=true , address=45.91.116.12:4258 , piece_id=KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:20.106010337 (601h42m19.977093372s) [noise=true , address=145.79.227.40:4053 , piece_id=JCSIUMWKEAQNQXTNSJTN6VSB3V2SIDRQK67QMK46B4GOUWEMQE2A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:20.106080911 (601h42m19.977163946s) [seg_pos={0 0}, upload=527370]: Segment reached optimal threshold of 14 pieces. Dec 30 17:11:20.106274520 (601h42m19.977357565s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Dec 30 17:11:20.106281644 (601h42m19.977364679s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold Dec 30 17:11:20.106351607 (601h42m19.977434642s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF Dec 30 17:11:20.106367767 (601h42m19.977450802s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold Dec 30 17:11:20.106424967 (601h42m19.977508002s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Dec 30 17:11:20.106432451 (601h42m19.977515486s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold Dec 30 17:11:20.106540666 (601h42m19.977623702s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 30 17:11:20.106546207 (601h42m19.977629242s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold Dec 30 17:11:20.106563380 (601h42m19.977646415s) [seg_pos={0 0}, upload=527370]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 17:11:20.106625529 (601h42m19.977708564s) [upload=527370]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 17:11:20.106629837 (601h42m19.977712882s) [upload=527370]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 17:11:20.406465452 (601h42m20.277548507s) [upload=527370]: Done uploading segments... Dec 30 17:11:20.406504206 (601h42m20.277587251s) [upload=527370]: Upload finished. err: <nil> Duration: 33.893774738s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X...X..X............................XX..................] duration: 10.256300649s trace: https://jaeger.dev.storj.io/trace/4173ea2fcd946b89 FAIL (2025-12-30 17:48:42 UTC) orbiter warning: context deadline exceeded Duration: 10.256359422s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [.X..X....X.XX..X..X......X.........X..XXX...................] duration: 3.683196801s trace: https://jaeger.dev.storj.io/trace/56692ebd5ab65c7c FAIL (2025-12-30 17:48:39 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.683238903s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 9.273189162s trace: https://jaeger.dev.storj.io/trace/6ea66b1ed6f4c6e5 FAIL (2025-12-30 17:48:20 UTC) orbiter warning: context deadline exceeded Duration: 9.273216735s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [X....X.XXXXX.XX.X.XXX......XX.XXX...XX..X.....X.XXX....XX...] duration: 5.976704329s trace: https://jaeger.dev.storj.io/trace/7499febccbc62e67 FAIL (2025-12-30 17:48:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 123269 (602h16m42.477206304s) [upload=527890]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 17:45:42.876710105 (602h16m42.74779315s) [seg_pos={0 0}, upload=527890]: Begin upload segment... Dec 30 17:45:42.876758677 (602h16m42.747841712s) [seg_pos={0 0}, upload=527890]: Done begin upload segment. Dec 30 17:45:42.876792852 (602h16m42.747875897s) [noise=true , address=45.91.116.12:4228 , piece_id=TEC23262YRTNV2WNAEBS5BZGTDQTMNSZCZG34OOR66HFKB5GQHHA, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.876848779 (602h16m42.747931824s) [noise=true , address=145.79.227.41:4087 , piece_id=JXD32DPHKLSQH36JRJZRPH6SI7ZZAMBL4AM64C6U6KSANSJWLZ3Q, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.877414867 (602h16m42.748497902s) [noise=true , address=145.79.227.3:4151 , piece_id=BLUNO6N5WXNON227NQB2MDJTF5X2OSHOW3EWHG4USXWJ5OMH2BCQ, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.877585782 (602h16m42.748668827s) [noise=true , address=45.91.116.4:4937 , piece_id=VN533XKJKWMBE5CT6N3BRY6UDZS43XCM44DMMR5JNJR5POKG6BJQ, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.877780573 (602h16m42.748863619s) [noise=true , address=145.79.227.12:4765 , piece_id=CTUCAP6LFRR4ZHSTI32NNWIUKCC6DIPWWTCI3Q55OYAUSJ4LQVKQ, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.877983619 (602h16m42.749066664s) [noise=true , address=145.79.227.8:4331 , piece_id=6GH4TYPK3CI3E3DKFRVN3FE6KGOPJ5QAWS44D5SV5F2GKGEEHQQA, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.878158562 (602h16m42.749241597s) [noise=true , address=145.79.227.7:4293 , piece_id=63XKHXW5V53VPRBL4ZE6LNPKFP3E4DX5VDZXQFLMDO2VGJT37M5Q, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.878398479 (602h16m42.749481524s) [noise=true , address=45.91.116.8:4046 , piece_id=A2VZG6A7PGGFNSEVMU2546FVCTLGM6UOYFV5B4SQ6LWECLD6MF2Q, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.878505303 (602h16m42.749588338s) [noise=true , address=145.79.227.51:4410 , piece_id=J7HM7D5JXY3LZBK4RKFOJ4XSNFWHHUNZAN4ZZRMFID2BZURDTXOA, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.878793262 (602h16m42.749876297s) [noise=true , address=145.79.227.39:4013 , piece_id=SAAZXLKDLXEY5QZEQANU7IFSE4E3NTC6N6XZJJ2C44EOW56OSJTA, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.878939831 (602h16m42.750022866s) [noise=true , address=45.91.116.11:4119 , piece_id=WYFKQ22R6IHWPJZ6EVH7IBFTIZCAR4ZRU75EUUBXMER2NIA2YKRA, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.879131355 (602h16m42.7502144s) [noise=true , address=145.79.227.25:4619 , piece_id=VNJVQJ3PVRV4FEPOTSNS3F5JXQ4GUX6VVDZT52GCZNEYNOQMHPMA, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.887768982 (602h16m42.758852017s) [noise=true , address=77.90.31.16:4735 , piece_id=BJGIVHNGAPFCJQH2NYOGUMNLQAA7APG4EVEPQNHIJNP4XHUHKQMQ, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.897314348 (602h16m42.768397383s) [noise=true , address=77.90.31.12:4590 , piece_id=PS4U5VCVVTOAJ4JGVDVOQIIOPS6MH2Y3KY724LKKK5KLUHADMKZQ, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.899145295 (602h16m42.77022833s) [noise=true , address=145.79.227.6:4257 , piece_id=C2V7ZWJZGPS7KZRU5T6UNMGOF5YWN6JTP6AUZ2E7I2H2ZS3HAAEA, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.899894471 (602h16m42.770977506s) [noise=true , address=145.79.227.21:4475 , piece_id=RB3FL546I4SVN3FRD7KL6FMYBX7D43WQYEAZUIMMRSJVZTHSDKLQ, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:44.520498726 (602h16m44.391581761s) [upload=527890]: Next returned nil segment Dec 30 17:45:44.520512822 (602h16m44.391595857s) [upload=527890]: Waiting for error group managing segments... Dec 30 17:45:44.862021472 (602h16m44.733104507s) [noise=true , address=145.79.227.7:4293 , piece_id=63XKHXW5V53VPRBL4ZE6LNPKFP3E4DX5VDZXQFLMDO2VGJT37M5Q, seg_pos={0 0}, upload=527890]: Done uploading piece. err: <nil> Dec 30 17:45:44.868822802 (602h16m44.739905848s) [noise=true , address=145.79.227.51:4410 , piece_id=J7HM7D5JXY3LZBK4RKFOJ4XSNFWHHUNZAN4ZZRMFID2BZURDTXOA, seg_pos={0 0}, upload=527890]: Done uploading piece. err: <nil> Dec 30 17:45:44.870439721 (602h16m44.741522756s) [noise=true , address=145.79.227.25:4619 , piece_id=VNJVQJ3PVRV4FEPOTSNS3F5JXQ4GUX6VVDZT52GCZNEYNOQMHPMA, seg_pos={0 0}, upload=527890]: Done uploading piece. err: <nil> Dec 30 17:45:44.871615740 (602h16m44.742698786s) [noise=true , address=145.79.227.39:4013 , piece_id=SAAZXLKDLXEY5QZEQANU7IFSE4E3NTC6N6XZJJ2C44EOW56OSJTA, seg_pos={0 0}, upload=527890]: Done uploading piece. err: <nil> Dec 30 17:45:44.883505831 (602h16m44.754588866s) [noise=true , address=145.79.227.12:4765 , piece_id=CTUCAP6LFRR4ZHSTI32NNWIUKCC6DIPWWTCI3Q55OYAUSJ4LQVKQ, seg_pos={0 0}, upload=527890]: Done uploading piece. err: <nil> Dec 30 17:45:44.889839371 (602h16m44.760922406s) [noise=true , address=145.79.227.8:4331 , piece_id=6GH4TYPK3CI3E3DKFRVN3FE6KGOPJ5QAWS44D5SV5F2GKGEEHQQA, seg_pos={0 0}, upload=527890]: Done uploading piece. err: <nil> Dec 30 17:45:44.950692544 (602h16m44.821775589s) [noise=true , address=145.79.227.41:4087 , piece_id=JXD32DPHKLSQH36JRJZRPH6SI7ZZAMBL4AM64C6U6KSANSJWLZ3Q, seg_pos={0 0}, upload=527890]: Done uploading piece. err: <nil> Dec 30 17:45:44.970768812 (602h16m44.841851847s) [noise=true , address=145.79.227.21:4475 , piece_id=RB3FL546I4SVN3FRD7KL6FMYBX7D43WQYEAZUIMMRSJVZTHSDKLQ, seg_pos={0 0}, upload=527890]: Done uploading piece. err: <nil> Dec 30 17:45:45.153744544 (602h16m45.024827579s) [noise=true , address=77.90.31.16:4735 , piece_id=BJGIVHNGAPFCJQH2NYOGUMNLQAA7APG4EVEPQNHIJNP4XHUHKQMQ, seg_pos={0 0}, upload=527890]: Done uploading piece. err: <nil> Dec 30 17:45:45.155910799 (602h16m45.026993844s) [noise=true , address=77.90.31.12:4590 , piece_id=PS4U5VCVVTOAJ4JGVDVOQIIOPS6MH2Y3KY724LKKK5KLUHADMKZQ, seg_pos={0 0}, upload=527890]: Done uploading piece. err: <nil> Dec 30 17:45:45.190278409 (602h16m45.061361464s) [noise=true , address=45.91.116.11:4119 , piece_id=WYFKQ22R6IHWPJZ6EVH7IBFTIZCAR4ZRU75EUUBXMER2NIA2YKRA, seg_pos={0 0}, upload=527890]: Done uploading piece. err: <nil> Dec 30 17:45:45.285402168 (602h16m45.156485203s) [noise=true , address=45.91.116.12:4228 , piece_id=TEC23262YRTNV2WNAEBS5BZGTDQTMNSZCZG34OOR66HFKB5GQHHA, seg_pos={0 0}, upload=527890]: Done uploading piece. err: <nil> Dec 30 17:45:45.958179638 (602h16m45.829262683s) [noise=true , address=45.91.116.4:4937 , piece_id=VN533XKJKWMBE5CT6N3BRY6UDZS43XCM44DMMR5JNJR5POKG6BJQ, seg_pos={0 0}, upload=527890]: Done uploading piece. err: <nil> Dec 30 17:45:45.958204887 (602h16m45.829287952s) [seg_pos={0 0}, upload=527890]: Segment reached optimal threshold of 13 pieces. Dec 30 17:45:45.958425907 (602h16m45.829508962s) [noise=true , address=45.91.116.8:4046 , piece_id=A2VZG6A7PGGFNSEVMU2546FVCTLGM6UOYFV5B4SQ6LWECLD6MF2Q, seg_pos={0 0}, upload=527890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: write tcp 188.166.225.170:51886->45.91.116.8:4046: use of closed network connection; write tcp 188.166.225.170:51886->45.91.116.8:4046: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51886->45.91.116.8:4046: use of closed network connection Dec 30 17:45:45.958435716 (602h16m45.829518771s) [noise=true , address=45.91.116.8:4046 , piece_id=A2VZG6A7PGGFNSEVMU2546FVCTLGM6UOYFV5B4SQ6LWECLD6MF2Q, seg_pos={0 0}, upload=527890]: Long tail cancellation due to optimal threshold Dec 30 17:45:45.958459822 (602h16m45.829542877s) [noise=true , address=145.79.227.6:4257 , piece_id=C2V7ZWJZGPS7KZRU5T6UNMGOF5YWN6JTP6AUZ2E7I2H2ZS3HAAEA, seg_pos={0 0}, upload=527890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:35142->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:35142->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35142->145.79.227.6:4257: use of closed network connection Dec 30 17:45:45.958469029 (602h16m45.829552094s) [noise=true , address=145.79.227.6:4257 , piece_id=C2V7ZWJZGPS7KZRU5T6UNMGOF5YWN6JTP6AUZ2E7I2H2ZS3HAAEA, seg_pos={0 0}, upload=527890]: Long tail cancellation due to optimal threshold Dec 30 17:45:45.958745265 (602h16m45.82982831s) [noise=true , address=145.79.227.3:4151 , piece_id=BLUNO6N5WXNON227NQB2MDJTF5X2OSHOW3EWHG4USXWJ5OMH2BCQ, seg_pos={0 0}, upload=527890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:48350->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:48350->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48350->145.79.227.3:4151: use of closed network connection Dec 30 17:45:45.958755043 (602h16m45.829838089s) [noise=true , address=145.79.227.3:4151 , piece_id=BLUNO6N5WXNON227NQB2MDJTF5X2OSHOW3EWHG4USXWJ5OMH2BCQ, seg_pos={0 0}, upload=527890]: Long tail cancellation due to optimal threshold Dec 30 17:45:45.958775753 (602h16m45.829858808s) [seg_pos={0 0}, upload=527890]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 17:45:45.958880272 (602h16m45.829963317s) [upload=527890]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 17:45:45.958887376 (602h16m45.829970411s) [upload=527890]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 17:45:46.257493423 (602h16m46.128576458s) [upload=527890]: Done uploading segments... Dec 30 17:45:46.257516557 (602h16m46.128599602s) [upload=527890]: Upload finished. err: <nil> Duration:* 5.976740919s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [..X..X....X.XX..X..X......X.........X..XXX..................] duration: 3.565141093s trace: https://jaeger.dev.storj.io/trace/56692ebd5ab65c7c FAIL (2025-12-30 17:48:08 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.565172144s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 35.745519489s trace: https://jaeger.dev.storj.io/trace/5ea8d7d4f61e13a2 FAIL (2025-12-30 17:48:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1X5A7bLqMEQuaMbf9xZ7PDUD4161sz2QqdM9r59y3bsyPBJhaK, Piece ID: HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ) context deadline exceeded; piecestore: (Node ID: 12eLKUg2vTnZMrLaraSPEoGXe3jDVBJYyURdcn4f2gCN5pvvCQ8, Piece ID: KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ) context deadline exceeded; piecestore: (Node ID: 12W4n6uf6azQk1QgeR8Xw7UkpnZm19zdyFKwojCsqZt2GFHrpce, Piece ID: DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ) context deadline exceeded ======LOGS====== 17.114181822 (601h42m16.985264867s) [noise=true , address=145.79.227.44:4208 , piece_id=3OXDCHZSOIFS5VQCI4SXSAQFICDCYHRE5NGFBERR33KOTGSZWLDQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.114641387 (601h42m16.985724432s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.115652532 (601h42m16.986735578s) [noise=true , address=77.90.31.17:4790 , piece_id=5RSRESKGDA2YF4NBRF4ZFABBKWJK4NO6VLEXW2Z6EY3OPXH2WPNA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.115937134 (601h42m16.987020179s) [noise=true , address=64.49.13.155:4923 , piece_id=MUJD7XLM2YRLRCHYVSDPBE6VBYLHCOA7HJQP22ZSET4N574XQMJA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.116814986 (601h42m16.987898031s) [noise=true , address=45.91.116.11:4139 , piece_id=72DPEWZ7ZPZZ53UGYV2VCUKLCK7HCM22CDBRHWVLJGDGO5B5JADQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.117501182 (601h42m16.988584228s) [noise=true , address=77.90.31.14:4665 , piece_id=DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.118263334 (601h42m16.989346379s) [noise=true , address=145.79.227.40:4084 , piece_id=ZADZDRKBIILLAI4IT335CYETOSFOFUUPQIH6HZJCTHKRTZSPDJUQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.118555850 (601h42m16.989638886s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.118765930 (601h42m16.989849036s) [noise=true , address=77.90.31.21:4905 , piece_id=HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.119029533 (601h42m16.990112568s) [noise=true , address=77.90.31.10:4533 , piece_id=QFBBYNQHQL2G7PEREK43FZBRUMKMVNBC2XKTE2NWTVTKN2452LPA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.119417672 (601h42m16.990500707s) [noise=true , address=45.91.116.12:4258 , piece_id=KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.119441588 (601h42m16.990524643s) [noise=true , address=145.79.227.40:4053 , piece_id=JCSIUMWKEAQNQXTNSJTN6VSB3V2SIDRQK67QMK46B4GOUWEMQE2A, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.119763470 (601h42m16.990846505s) [noise=true , address=77.90.31.18:4821 , piece_id=MXZFKXM3JV2HQURDLIU7BKWLAKDZ27O4ER3SGC7QBZV5I3J2EHKA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.120028685 (601h42m16.99111172s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.120086986 (601h42m16.991170031s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.120586507 (601h42m16.991669562s) [noise=true , address=45.91.116.13:4168 , piece_id=AKUE5H3A2UECQOZY3GV3MWZKIHSNKCDOG32DKCNHIRTJROJ74M3A, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.121258768 (601h42m16.992341813s) [noise=true , address=145.79.227.9:4379 , piece_id=QTFXFSGGKYBQSPWJYBBJMQUL3D5EEO3IX4C53ECD7FXMUOGUBC7Q, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:18.568739165 (601h42m18.43982221s) [upload=527370]: Next returned nil segment Dec 30 17:11:18.568751328 (601h42m18.439834363s) [upload=527370]: Waiting for error group managing segments... Dec 30 17:11:18.978424178 (601h42m18.849507213s) [noise=true , address=64.49.13.155:4923 , piece_id=MUJD7XLM2YRLRCHYVSDPBE6VBYLHCOA7HJQP22ZSET4N574XQMJA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:18.986258125 (601h42m18.85734116s) [noise=true , address=145.79.227.9:4379 , piece_id=QTFXFSGGKYBQSPWJYBBJMQUL3D5EEO3IX4C53ECD7FXMUOGUBC7Q, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.274519421 (601h42m19.145602456s) [noise=true , address=77.90.31.14:4665 , piece_id=DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.315775438 (601h42m19.186858473s) [noise=true , address=45.91.116.13:4168 , piece_id=AKUE5H3A2UECQOZY3GV3MWZKIHSNKCDOG32DKCNHIRTJROJ74M3A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.335765371 (601h42m19.206848406s) [noise=true , address=77.90.31.21:4905 , piece_id=HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.354812129 (601h42m19.225895164s) [noise=true , address=77.90.31.10:4533 , piece_id=QFBBYNQHQL2G7PEREK43FZBRUMKMVNBC2XKTE2NWTVTKN2452LPA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.392024917 (601h42m19.263107952s) [noise=true , address=45.91.116.11:4139 , piece_id=72DPEWZ7ZPZZ53UGYV2VCUKLCK7HCM22CDBRHWVLJGDGO5B5JADQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.396084327 (601h42m19.267167372s) [noise=true , address=77.90.31.18:4821 , piece_id=MXZFKXM3JV2HQURDLIU7BKWLAKDZ27O4ER3SGC7QBZV5I3J2EHKA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.862099099 (601h42m19.733182134s) [noise=true , address=77.90.31.17:4790 , piece_id=5RSRESKGDA2YF4NBRF4ZFABBKWJK4NO6VLEXW2Z6EY3OPXH2WPNA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.905412913 (601h42m19.776495948s) [noise=true , address=145.79.227.44:4208 , piece_id=3OXDCHZSOIFS5VQCI4SXSAQFICDCYHRE5NGFBERR33KOTGSZWLDQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.910309688 (601h42m19.781392743s) [noise=true , address=145.79.227.30:4794 , piece_id=EJH6Q3T2BMPVLG6F7CW7N4GYOA3EZFPADL6VPT7WKKG65B2WZVXQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.934233361 (601h42m19.805316397s) [noise=true , address=145.79.227.40:4084 , piece_id=ZADZDRKBIILLAI4IT335CYETOSFOFUUPQIH6HZJCTHKRTZSPDJUQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:20.074011527 (601h42m19.945094572s) [noise=true , address=45.91.116.12:4258 , piece_id=KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:20.106010337 (601h42m19.977093372s) [noise=true , address=145.79.227.40:4053 , piece_id=JCSIUMWKEAQNQXTNSJTN6VSB3V2SIDRQK67QMK46B4GOUWEMQE2A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:20.106080911 (601h42m19.977163946s) [seg_pos={0 0}, upload=527370]: Segment reached optimal threshold of 14 pieces. Dec 30 17:11:20.106274520 (601h42m19.977357565s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Dec 30 17:11:20.106281644 (601h42m19.977364679s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold Dec 30 17:11:20.106351607 (601h42m19.977434642s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF Dec 30 17:11:20.106367767 (601h42m19.977450802s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold Dec 30 17:11:20.106424967 (601h42m19.977508002s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Dec 30 17:11:20.106432451 (601h42m19.977515486s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold Dec 30 17:11:20.106540666 (601h42m19.977623702s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 30 17:11:20.106546207 (601h42m19.977629242s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold Dec 30 17:11:20.106563380 (601h42m19.977646415s) [seg_pos={0 0}, upload=527370]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 17:11:20.106625529 (601h42m19.977708564s) [upload=527370]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 17:11:20.106629837 (601h42m19.977712882s) [upload=527370]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 17:11:20.406465452 (601h42m20.277548507s) [upload=527370]: Done uploading segments... Dec 30 17:11:20.406504206 (601h42m20.277587251s) [upload=527370]: Upload finished. err: <nil> Duration: 35.745534175s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [.X....X.XXXXX.XX.X.XXX......XX.XXX...XX..X.....X.XXX....XX..] duration: 6.253287283s trace: https://jaeger.dev.storj.io/trace/7499febccbc62e67 FAIL (2025-12-30 17:47:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 123269 (602h16m42.477206304s) [upload=527890]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 17:45:42.876710105 (602h16m42.74779315s) [seg_pos={0 0}, upload=527890]: Begin upload segment... Dec 30 17:45:42.876758677 (602h16m42.747841712s) [seg_pos={0 0}, upload=527890]: Done begin upload segment. Dec 30 17:45:42.876792852 (602h16m42.747875897s) [noise=true , address=45.91.116.12:4228 , piece_id=TEC23262YRTNV2WNAEBS5BZGTDQTMNSZCZG34OOR66HFKB5GQHHA, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.876848779 (602h16m42.747931824s) [noise=true , address=145.79.227.41:4087 , piece_id=JXD32DPHKLSQH36JRJZRPH6SI7ZZAMBL4AM64C6U6KSANSJWLZ3Q, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.877414867 (602h16m42.748497902s) [noise=true , address=145.79.227.3:4151 , piece_id=BLUNO6N5WXNON227NQB2MDJTF5X2OSHOW3EWHG4USXWJ5OMH2BCQ, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.877585782 (602h16m42.748668827s) [noise=true , address=45.91.116.4:4937 , piece_id=VN533XKJKWMBE5CT6N3BRY6UDZS43XCM44DMMR5JNJR5POKG6BJQ, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.877780573 (602h16m42.748863619s) [noise=true , address=145.79.227.12:4765 , piece_id=CTUCAP6LFRR4ZHSTI32NNWIUKCC6DIPWWTCI3Q55OYAUSJ4LQVKQ, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.877983619 (602h16m42.749066664s) [noise=true , address=145.79.227.8:4331 , piece_id=6GH4TYPK3CI3E3DKFRVN3FE6KGOPJ5QAWS44D5SV5F2GKGEEHQQA, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.878158562 (602h16m42.749241597s) [noise=true , address=145.79.227.7:4293 , piece_id=63XKHXW5V53VPRBL4ZE6LNPKFP3E4DX5VDZXQFLMDO2VGJT37M5Q, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.878398479 (602h16m42.749481524s) [noise=true , address=45.91.116.8:4046 , piece_id=A2VZG6A7PGGFNSEVMU2546FVCTLGM6UOYFV5B4SQ6LWECLD6MF2Q, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.878505303 (602h16m42.749588338s) [noise=true , address=145.79.227.51:4410 , piece_id=J7HM7D5JXY3LZBK4RKFOJ4XSNFWHHUNZAN4ZZRMFID2BZURDTXOA, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.878793262 (602h16m42.749876297s) [noise=true , address=145.79.227.39:4013 , piece_id=SAAZXLKDLXEY5QZEQANU7IFSE4E3NTC6N6XZJJ2C44EOW56OSJTA, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.878939831 (602h16m42.750022866s) [noise=true , address=45.91.116.11:4119 , piece_id=WYFKQ22R6IHWPJZ6EVH7IBFTIZCAR4ZRU75EUUBXMER2NIA2YKRA, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.879131355 (602h16m42.7502144s) [noise=true , address=145.79.227.25:4619 , piece_id=VNJVQJ3PVRV4FEPOTSNS3F5JXQ4GUX6VVDZT52GCZNEYNOQMHPMA, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.887768982 (602h16m42.758852017s) [noise=true , address=77.90.31.16:4735 , piece_id=BJGIVHNGAPFCJQH2NYOGUMNLQAA7APG4EVEPQNHIJNP4XHUHKQMQ, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.897314348 (602h16m42.768397383s) [noise=true , address=77.90.31.12:4590 , piece_id=PS4U5VCVVTOAJ4JGVDVOQIIOPS6MH2Y3KY724LKKK5KLUHADMKZQ, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.899145295 (602h16m42.77022833s) [noise=true , address=145.79.227.6:4257 , piece_id=C2V7ZWJZGPS7KZRU5T6UNMGOF5YWN6JTP6AUZ2E7I2H2ZS3HAAEA, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.899894471 (602h16m42.770977506s) [noise=true , address=145.79.227.21:4475 , piece_id=RB3FL546I4SVN3FRD7KL6FMYBX7D43WQYEAZUIMMRSJVZTHSDKLQ, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:44.520498726 (602h16m44.391581761s) [upload=527890]: Next returned nil segment Dec 30 17:45:44.520512822 (602h16m44.391595857s) [upload=527890]: Waiting for error group managing segments... Dec 30 17:45:44.862021472 (602h16m44.733104507s) [noise=true , address=145.79.227.7:4293 , piece_id=63XKHXW5V53VPRBL4ZE6LNPKFP3E4DX5VDZXQFLMDO2VGJT37M5Q, seg_pos={0 0}, upload=527890]: Done uploading piece. err: <nil> Dec 30 17:45:44.868822802 (602h16m44.739905848s) [noise=true , address=145.79.227.51:4410 , piece_id=J7HM7D5JXY3LZBK4RKFOJ4XSNFWHHUNZAN4ZZRMFID2BZURDTXOA, seg_pos={0 0}, upload=527890]: Done uploading piece. err: <nil> Dec 30 17:45:44.870439721 (602h16m44.741522756s) [noise=true , address=145.79.227.25:4619 , piece_id=VNJVQJ3PVRV4FEPOTSNS3F5JXQ4GUX6VVDZT52GCZNEYNOQMHPMA, seg_pos={0 0}, upload=527890]: Done uploading piece. err: <nil> Dec 30 17:45:44.871615740 (602h16m44.742698786s) [noise=true , address=145.79.227.39:4013 , piece_id=SAAZXLKDLXEY5QZEQANU7IFSE4E3NTC6N6XZJJ2C44EOW56OSJTA, seg_pos={0 0}, upload=527890]: Done uploading piece. err: <nil> Dec 30 17:45:44.883505831 (602h16m44.754588866s) [noise=true , address=145.79.227.12:4765 , piece_id=CTUCAP6LFRR4ZHSTI32NNWIUKCC6DIPWWTCI3Q55OYAUSJ4LQVKQ, seg_pos={0 0}, upload=527890]: Done uploading piece. err: <nil> Dec 30 17:45:44.889839371 (602h16m44.760922406s) [noise=true , address=145.79.227.8:4331 , piece_id=6GH4TYPK3CI3E3DKFRVN3FE6KGOPJ5QAWS44D5SV5F2GKGEEHQQA, seg_pos={0 0}, upload=527890]: Done uploading piece. err: <nil> Dec 30 17:45:44.950692544 (602h16m44.821775589s) [noise=true , address=145.79.227.41:4087 , piece_id=JXD32DPHKLSQH36JRJZRPH6SI7ZZAMBL4AM64C6U6KSANSJWLZ3Q, seg_pos={0 0}, upload=527890]: Done uploading piece. err: <nil> Dec 30 17:45:44.970768812 (602h16m44.841851847s) [noise=true , address=145.79.227.21:4475 , piece_id=RB3FL546I4SVN3FRD7KL6FMYBX7D43WQYEAZUIMMRSJVZTHSDKLQ, seg_pos={0 0}, upload=527890]: Done uploading piece. err: <nil> Dec 30 17:45:45.153744544 (602h16m45.024827579s) [noise=true , address=77.90.31.16:4735 , piece_id=BJGIVHNGAPFCJQH2NYOGUMNLQAA7APG4EVEPQNHIJNP4XHUHKQMQ, seg_pos={0 0}, upload=527890]: Done uploading piece. err: <nil> Dec 30 17:45:45.155910799 (602h16m45.026993844s) [noise=true , address=77.90.31.12:4590 , piece_id=PS4U5VCVVTOAJ4JGVDVOQIIOPS6MH2Y3KY724LKKK5KLUHADMKZQ, seg_pos={0 0}, upload=527890]: Done uploading piece. err: <nil> Dec 30 17:45:45.190278409 (602h16m45.061361464s) [noise=true , address=45.91.116.11:4119 , piece_id=WYFKQ22R6IHWPJZ6EVH7IBFTIZCAR4ZRU75EUUBXMER2NIA2YKRA, seg_pos={0 0}, upload=527890]: Done uploading piece. err: <nil> Dec 30 17:45:45.285402168 (602h16m45.156485203s) [noise=true , address=45.91.116.12:4228 , piece_id=TEC23262YRTNV2WNAEBS5BZGTDQTMNSZCZG34OOR66HFKB5GQHHA, seg_pos={0 0}, upload=527890]: Done uploading piece. err: <nil> Dec 30 17:45:45.958179638 (602h16m45.829262683s) [noise=true , address=45.91.116.4:4937 , piece_id=VN533XKJKWMBE5CT6N3BRY6UDZS43XCM44DMMR5JNJR5POKG6BJQ, seg_pos={0 0}, upload=527890]: Done uploading piece. err: <nil> Dec 30 17:45:45.958204887 (602h16m45.829287952s) [seg_pos={0 0}, upload=527890]: Segment reached optimal threshold of 13 pieces. Dec 30 17:45:45.958425907 (602h16m45.829508962s) [noise=true , address=45.91.116.8:4046 , piece_id=A2VZG6A7PGGFNSEVMU2546FVCTLGM6UOYFV5B4SQ6LWECLD6MF2Q, seg_pos={0 0}, upload=527890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: write tcp 188.166.225.170:51886->45.91.116.8:4046: use of closed network connection; write tcp 188.166.225.170:51886->45.91.116.8:4046: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51886->45.91.116.8:4046: use of closed network connection Dec 30 17:45:45.958435716 (602h16m45.829518771s) [noise=true , address=45.91.116.8:4046 , piece_id=A2VZG6A7PGGFNSEVMU2546FVCTLGM6UOYFV5B4SQ6LWECLD6MF2Q, seg_pos={0 0}, upload=527890]: Long tail cancellation due to optimal threshold Dec 30 17:45:45.958459822 (602h16m45.829542877s) [noise=true , address=145.79.227.6:4257 , piece_id=C2V7ZWJZGPS7KZRU5T6UNMGOF5YWN6JTP6AUZ2E7I2H2ZS3HAAEA, seg_pos={0 0}, upload=527890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:35142->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:35142->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35142->145.79.227.6:4257: use of closed network connection Dec 30 17:45:45.958469029 (602h16m45.829552094s) [noise=true , address=145.79.227.6:4257 , piece_id=C2V7ZWJZGPS7KZRU5T6UNMGOF5YWN6JTP6AUZ2E7I2H2ZS3HAAEA, seg_pos={0 0}, upload=527890]: Long tail cancellation due to optimal threshold Dec 30 17:45:45.958745265 (602h16m45.82982831s) [noise=true , address=145.79.227.3:4151 , piece_id=BLUNO6N5WXNON227NQB2MDJTF5X2OSHOW3EWHG4USXWJ5OMH2BCQ, seg_pos={0 0}, upload=527890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:48350->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:48350->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48350->145.79.227.3:4151: use of closed network connection Dec 30 17:45:45.958755043 (602h16m45.829838089s) [noise=true , address=145.79.227.3:4151 , piece_id=BLUNO6N5WXNON227NQB2MDJTF5X2OSHOW3EWHG4USXWJ5OMH2BCQ, seg_pos={0 0}, upload=527890]: Long tail cancellation due to optimal threshold Dec 30 17:45:45.958775753 (602h16m45.829858808s) [seg_pos={0 0}, upload=527890]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 17:45:45.958880272 (602h16m45.829963317s) [upload=527890]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 17:45:45.958887376 (602h16m45.829970411s) [upload=527890]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 17:45:46.257493423 (602h16m46.128576458s) [upload=527890]: Done uploading segments... Dec 30 17:45:46.257516557 (602h16m46.128599602s) [upload=527890]: Upload finished. err: <nil> Duration:* 6.253315768s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....X...X..X............................XX.................] duration: 6.209872012s trace: https://jaeger.dev.storj.io/trace/4173ea2fcd946b89 FAIL (2025-12-30 17:47:57 UTC) orbiter warning: context deadline exceeded Duration: 6.20990792s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 11.55670493s trace: https://jaeger.dev.storj.io/trace/6ea66b1ed6f4c6e5 FAIL (2025-12-30 17:47:46 UTC) orbiter warning: context deadline exceeded Duration: 11.55672036s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 37.610852674s trace: https://jaeger.dev.storj.io/trace/5ea8d7d4f61e13a2 FAIL (2025-12-30 17:47:30 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1X5A7bLqMEQuaMbf9xZ7PDUD4161sz2QqdM9r59y3bsyPBJhaK, Piece ID: HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ) context deadline exceeded; piecestore: (Node ID: 12eLKUg2vTnZMrLaraSPEoGXe3jDVBJYyURdcn4f2gCN5pvvCQ8, Piece ID: KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ) context deadline exceeded; piecestore: (Node ID: 12W4n6uf6azQk1QgeR8Xw7UkpnZm19zdyFKwojCsqZt2GFHrpce, Piece ID: DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ) context deadline exceeded ======LOGS====== 17.114181822 (601h42m16.985264867s) [noise=true , address=145.79.227.44:4208 , piece_id=3OXDCHZSOIFS5VQCI4SXSAQFICDCYHRE5NGFBERR33KOTGSZWLDQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.114641387 (601h42m16.985724432s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.115652532 (601h42m16.986735578s) [noise=true , address=77.90.31.17:4790 , piece_id=5RSRESKGDA2YF4NBRF4ZFABBKWJK4NO6VLEXW2Z6EY3OPXH2WPNA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.115937134 (601h42m16.987020179s) [noise=true , address=64.49.13.155:4923 , piece_id=MUJD7XLM2YRLRCHYVSDPBE6VBYLHCOA7HJQP22ZSET4N574XQMJA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.116814986 (601h42m16.987898031s) [noise=true , address=45.91.116.11:4139 , piece_id=72DPEWZ7ZPZZ53UGYV2VCUKLCK7HCM22CDBRHWVLJGDGO5B5JADQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.117501182 (601h42m16.988584228s) [noise=true , address=77.90.31.14:4665 , piece_id=DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.118263334 (601h42m16.989346379s) [noise=true , address=145.79.227.40:4084 , piece_id=ZADZDRKBIILLAI4IT335CYETOSFOFUUPQIH6HZJCTHKRTZSPDJUQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.118555850 (601h42m16.989638886s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.118765930 (601h42m16.989849036s) [noise=true , address=77.90.31.21:4905 , piece_id=HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.119029533 (601h42m16.990112568s) [noise=true , address=77.90.31.10:4533 , piece_id=QFBBYNQHQL2G7PEREK43FZBRUMKMVNBC2XKTE2NWTVTKN2452LPA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.119417672 (601h42m16.990500707s) [noise=true , address=45.91.116.12:4258 , piece_id=KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.119441588 (601h42m16.990524643s) [noise=true , address=145.79.227.40:4053 , piece_id=JCSIUMWKEAQNQXTNSJTN6VSB3V2SIDRQK67QMK46B4GOUWEMQE2A, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.119763470 (601h42m16.990846505s) [noise=true , address=77.90.31.18:4821 , piece_id=MXZFKXM3JV2HQURDLIU7BKWLAKDZ27O4ER3SGC7QBZV5I3J2EHKA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.120028685 (601h42m16.99111172s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.120086986 (601h42m16.991170031s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.120586507 (601h42m16.991669562s) [noise=true , address=45.91.116.13:4168 , piece_id=AKUE5H3A2UECQOZY3GV3MWZKIHSNKCDOG32DKCNHIRTJROJ74M3A, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.121258768 (601h42m16.992341813s) [noise=true , address=145.79.227.9:4379 , piece_id=QTFXFSGGKYBQSPWJYBBJMQUL3D5EEO3IX4C53ECD7FXMUOGUBC7Q, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:18.568739165 (601h42m18.43982221s) [upload=527370]: Next returned nil segment Dec 30 17:11:18.568751328 (601h42m18.439834363s) [upload=527370]: Waiting for error group managing segments... Dec 30 17:11:18.978424178 (601h42m18.849507213s) [noise=true , address=64.49.13.155:4923 , piece_id=MUJD7XLM2YRLRCHYVSDPBE6VBYLHCOA7HJQP22ZSET4N574XQMJA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:18.986258125 (601h42m18.85734116s) [noise=true , address=145.79.227.9:4379 , piece_id=QTFXFSGGKYBQSPWJYBBJMQUL3D5EEO3IX4C53ECD7FXMUOGUBC7Q, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.274519421 (601h42m19.145602456s) [noise=true , address=77.90.31.14:4665 , piece_id=DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.315775438 (601h42m19.186858473s) [noise=true , address=45.91.116.13:4168 , piece_id=AKUE5H3A2UECQOZY3GV3MWZKIHSNKCDOG32DKCNHIRTJROJ74M3A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.335765371 (601h42m19.206848406s) [noise=true , address=77.90.31.21:4905 , piece_id=HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.354812129 (601h42m19.225895164s) [noise=true , address=77.90.31.10:4533 , piece_id=QFBBYNQHQL2G7PEREK43FZBRUMKMVNBC2XKTE2NWTVTKN2452LPA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.392024917 (601h42m19.263107952s) [noise=true , address=45.91.116.11:4139 , piece_id=72DPEWZ7ZPZZ53UGYV2VCUKLCK7HCM22CDBRHWVLJGDGO5B5JADQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.396084327 (601h42m19.267167372s) [noise=true , address=77.90.31.18:4821 , piece_id=MXZFKXM3JV2HQURDLIU7BKWLAKDZ27O4ER3SGC7QBZV5I3J2EHKA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.862099099 (601h42m19.733182134s) [noise=true , address=77.90.31.17:4790 , piece_id=5RSRESKGDA2YF4NBRF4ZFABBKWJK4NO6VLEXW2Z6EY3OPXH2WPNA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.905412913 (601h42m19.776495948s) [noise=true , address=145.79.227.44:4208 , piece_id=3OXDCHZSOIFS5VQCI4SXSAQFICDCYHRE5NGFBERR33KOTGSZWLDQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.910309688 (601h42m19.781392743s) [noise=true , address=145.79.227.30:4794 , piece_id=EJH6Q3T2BMPVLG6F7CW7N4GYOA3EZFPADL6VPT7WKKG65B2WZVXQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.934233361 (601h42m19.805316397s) [noise=true , address=145.79.227.40:4084 , piece_id=ZADZDRKBIILLAI4IT335CYETOSFOFUUPQIH6HZJCTHKRTZSPDJUQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:20.074011527 (601h42m19.945094572s) [noise=true , address=45.91.116.12:4258 , piece_id=KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:20.106010337 (601h42m19.977093372s) [noise=true , address=145.79.227.40:4053 , piece_id=JCSIUMWKEAQNQXTNSJTN6VSB3V2SIDRQK67QMK46B4GOUWEMQE2A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:20.106080911 (601h42m19.977163946s) [seg_pos={0 0}, upload=527370]: Segment reached optimal threshold of 14 pieces. Dec 30 17:11:20.106274520 (601h42m19.977357565s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Dec 30 17:11:20.106281644 (601h42m19.977364679s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold Dec 30 17:11:20.106351607 (601h42m19.977434642s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF Dec 30 17:11:20.106367767 (601h42m19.977450802s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold Dec 30 17:11:20.106424967 (601h42m19.977508002s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Dec 30 17:11:20.106432451 (601h42m19.977515486s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold Dec 30 17:11:20.106540666 (601h42m19.977623702s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 30 17:11:20.106546207 (601h42m19.977629242s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold Dec 30 17:11:20.106563380 (601h42m19.977646415s) [seg_pos={0 0}, upload=527370]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 17:11:20.106625529 (601h42m19.977708564s) [upload=527370]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 17:11:20.106629837 (601h42m19.977712882s) [upload=527370]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 17:11:20.406465452 (601h42m20.277548507s) [upload=527370]: Done uploading segments... Dec 30 17:11:20.406504206 (601h42m20.277587251s) [upload=527370]: Upload finished. err: <nil> Duration: 37.610873864s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [...X..X....X.XX..X..X......X.........X..XXX.................] duration: 3.410086508s trace: https://jaeger.dev.storj.io/trace/56692ebd5ab65c7c FAIL (2025-12-30 17:47:18 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.410109811s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.912177472s trace: https://jaeger.dev.storj.io/trace/6ea66b1ed6f4c6e5 FAIL (2025-12-30 17:47:11 UTC) orbiter warning: context deadline exceeded Duration: 5.912210295s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......X...X..X............................XX................] duration: 12.691633414s trace: https://jaeger.dev.storj.io/trace/4173ea2fcd946b89 FAIL (2025-12-30 17:46:59 UTC) orbiter warning: context deadline exceeded Duration: 12.691658481s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [X.X....X.XXXXX.XX.X.XXX......XX.XXX...XX..X.....X.XXX....XX.] duration: 5.811970525s trace: https://jaeger.dev.storj.io/trace/7499febccbc62e67 FAIL (2025-12-30 17:46:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 123269 (602h16m42.477206304s) [upload=527890]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 17:45:42.876710105 (602h16m42.74779315s) [seg_pos={0 0}, upload=527890]: Begin upload segment... Dec 30 17:45:42.876758677 (602h16m42.747841712s) [seg_pos={0 0}, upload=527890]: Done begin upload segment. Dec 30 17:45:42.876792852 (602h16m42.747875897s) [noise=true , address=45.91.116.12:4228 , piece_id=TEC23262YRTNV2WNAEBS5BZGTDQTMNSZCZG34OOR66HFKB5GQHHA, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.876848779 (602h16m42.747931824s) [noise=true , address=145.79.227.41:4087 , piece_id=JXD32DPHKLSQH36JRJZRPH6SI7ZZAMBL4AM64C6U6KSANSJWLZ3Q, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.877414867 (602h16m42.748497902s) [noise=true , address=145.79.227.3:4151 , piece_id=BLUNO6N5WXNON227NQB2MDJTF5X2OSHOW3EWHG4USXWJ5OMH2BCQ, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.877585782 (602h16m42.748668827s) [noise=true , address=45.91.116.4:4937 , piece_id=VN533XKJKWMBE5CT6N3BRY6UDZS43XCM44DMMR5JNJR5POKG6BJQ, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.877780573 (602h16m42.748863619s) [noise=true , address=145.79.227.12:4765 , piece_id=CTUCAP6LFRR4ZHSTI32NNWIUKCC6DIPWWTCI3Q55OYAUSJ4LQVKQ, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.877983619 (602h16m42.749066664s) [noise=true , address=145.79.227.8:4331 , piece_id=6GH4TYPK3CI3E3DKFRVN3FE6KGOPJ5QAWS44D5SV5F2GKGEEHQQA, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.878158562 (602h16m42.749241597s) [noise=true , address=145.79.227.7:4293 , piece_id=63XKHXW5V53VPRBL4ZE6LNPKFP3E4DX5VDZXQFLMDO2VGJT37M5Q, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.878398479 (602h16m42.749481524s) [noise=true , address=45.91.116.8:4046 , piece_id=A2VZG6A7PGGFNSEVMU2546FVCTLGM6UOYFV5B4SQ6LWECLD6MF2Q, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.878505303 (602h16m42.749588338s) [noise=true , address=145.79.227.51:4410 , piece_id=J7HM7D5JXY3LZBK4RKFOJ4XSNFWHHUNZAN4ZZRMFID2BZURDTXOA, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.878793262 (602h16m42.749876297s) [noise=true , address=145.79.227.39:4013 , piece_id=SAAZXLKDLXEY5QZEQANU7IFSE4E3NTC6N6XZJJ2C44EOW56OSJTA, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.878939831 (602h16m42.750022866s) [noise=true , address=45.91.116.11:4119 , piece_id=WYFKQ22R6IHWPJZ6EVH7IBFTIZCAR4ZRU75EUUBXMER2NIA2YKRA, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.879131355 (602h16m42.7502144s) [noise=true , address=145.79.227.25:4619 , piece_id=VNJVQJ3PVRV4FEPOTSNS3F5JXQ4GUX6VVDZT52GCZNEYNOQMHPMA, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.887768982 (602h16m42.758852017s) [noise=true , address=77.90.31.16:4735 , piece_id=BJGIVHNGAPFCJQH2NYOGUMNLQAA7APG4EVEPQNHIJNP4XHUHKQMQ, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.897314348 (602h16m42.768397383s) [noise=true , address=77.90.31.12:4590 , piece_id=PS4U5VCVVTOAJ4JGVDVOQIIOPS6MH2Y3KY724LKKK5KLUHADMKZQ, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.899145295 (602h16m42.77022833s) [noise=true , address=145.79.227.6:4257 , piece_id=C2V7ZWJZGPS7KZRU5T6UNMGOF5YWN6JTP6AUZ2E7I2H2ZS3HAAEA, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.899894471 (602h16m42.770977506s) [noise=true , address=145.79.227.21:4475 , piece_id=RB3FL546I4SVN3FRD7KL6FMYBX7D43WQYEAZUIMMRSJVZTHSDKLQ, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:44.520498726 (602h16m44.391581761s) [upload=527890]: Next returned nil segment Dec 30 17:45:44.520512822 (602h16m44.391595857s) [upload=527890]: Waiting for error group managing segments... Dec 30 17:45:44.862021472 (602h16m44.733104507s) [noise=true , address=145.79.227.7:4293 , piece_id=63XKHXW5V53VPRBL4ZE6LNPKFP3E4DX5VDZXQFLMDO2VGJT37M5Q, seg_pos={0 0}, upload=527890]: Done uploading piece. err: <nil> Dec 30 17:45:44.868822802 (602h16m44.739905848s) [noise=true , address=145.79.227.51:4410 , piece_id=J7HM7D5JXY3LZBK4RKFOJ4XSNFWHHUNZAN4ZZRMFID2BZURDTXOA, seg_pos={0 0}, upload=527890]: Done uploading piece. err: <nil> Dec 30 17:45:44.870439721 (602h16m44.741522756s) [noise=true , address=145.79.227.25:4619 , piece_id=VNJVQJ3PVRV4FEPOTSNS3F5JXQ4GUX6VVDZT52GCZNEYNOQMHPMA, seg_pos={0 0}, upload=527890]: Done uploading piece. err: <nil> Dec 30 17:45:44.871615740 (602h16m44.742698786s) [noise=true , address=145.79.227.39:4013 , piece_id=SAAZXLKDLXEY5QZEQANU7IFSE4E3NTC6N6XZJJ2C44EOW56OSJTA, seg_pos={0 0}, upload=527890]: Done uploading piece. err: <nil> Dec 30 17:45:44.883505831 (602h16m44.754588866s) [noise=true , address=145.79.227.12:4765 , piece_id=CTUCAP6LFRR4ZHSTI32NNWIUKCC6DIPWWTCI3Q55OYAUSJ4LQVKQ, seg_pos={0 0}, upload=527890]: Done uploading piece. err: <nil> Dec 30 17:45:44.889839371 (602h16m44.760922406s) [noise=true , address=145.79.227.8:4331 , piece_id=6GH4TYPK3CI3E3DKFRVN3FE6KGOPJ5QAWS44D5SV5F2GKGEEHQQA, seg_pos={0 0}, upload=527890]: Done uploading piece. err: <nil> Dec 30 17:45:44.950692544 (602h16m44.821775589s) [noise=true , address=145.79.227.41:4087 , piece_id=JXD32DPHKLSQH36JRJZRPH6SI7ZZAMBL4AM64C6U6KSANSJWLZ3Q, seg_pos={0 0}, upload=527890]: Done uploading piece. err: <nil> Dec 30 17:45:44.970768812 (602h16m44.841851847s) [noise=true , address=145.79.227.21:4475 , piece_id=RB3FL546I4SVN3FRD7KL6FMYBX7D43WQYEAZUIMMRSJVZTHSDKLQ, seg_pos={0 0}, upload=527890]: Done uploading piece. err: <nil> Dec 30 17:45:45.153744544 (602h16m45.024827579s) [noise=true , address=77.90.31.16:4735 , piece_id=BJGIVHNGAPFCJQH2NYOGUMNLQAA7APG4EVEPQNHIJNP4XHUHKQMQ, seg_pos={0 0}, upload=527890]: Done uploading piece. err: <nil> Dec 30 17:45:45.155910799 (602h16m45.026993844s) [noise=true , address=77.90.31.12:4590 , piece_id=PS4U5VCVVTOAJ4JGVDVOQIIOPS6MH2Y3KY724LKKK5KLUHADMKZQ, seg_pos={0 0}, upload=527890]: Done uploading piece. err: <nil> Dec 30 17:45:45.190278409 (602h16m45.061361464s) [noise=true , address=45.91.116.11:4119 , piece_id=WYFKQ22R6IHWPJZ6EVH7IBFTIZCAR4ZRU75EUUBXMER2NIA2YKRA, seg_pos={0 0}, upload=527890]: Done uploading piece. err: <nil> Dec 30 17:45:45.285402168 (602h16m45.156485203s) [noise=true , address=45.91.116.12:4228 , piece_id=TEC23262YRTNV2WNAEBS5BZGTDQTMNSZCZG34OOR66HFKB5GQHHA, seg_pos={0 0}, upload=527890]: Done uploading piece. err: <nil> Dec 30 17:45:45.958179638 (602h16m45.829262683s) [noise=true , address=45.91.116.4:4937 , piece_id=VN533XKJKWMBE5CT6N3BRY6UDZS43XCM44DMMR5JNJR5POKG6BJQ, seg_pos={0 0}, upload=527890]: Done uploading piece. err: <nil> Dec 30 17:45:45.958204887 (602h16m45.829287952s) [seg_pos={0 0}, upload=527890]: Segment reached optimal threshold of 13 pieces. Dec 30 17:45:45.958425907 (602h16m45.829508962s) [noise=true , address=45.91.116.8:4046 , piece_id=A2VZG6A7PGGFNSEVMU2546FVCTLGM6UOYFV5B4SQ6LWECLD6MF2Q, seg_pos={0 0}, upload=527890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: write tcp 188.166.225.170:51886->45.91.116.8:4046: use of closed network connection; write tcp 188.166.225.170:51886->45.91.116.8:4046: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51886->45.91.116.8:4046: use of closed network connection Dec 30 17:45:45.958435716 (602h16m45.829518771s) [noise=true , address=45.91.116.8:4046 , piece_id=A2VZG6A7PGGFNSEVMU2546FVCTLGM6UOYFV5B4SQ6LWECLD6MF2Q, seg_pos={0 0}, upload=527890]: Long tail cancellation due to optimal threshold Dec 30 17:45:45.958459822 (602h16m45.829542877s) [noise=true , address=145.79.227.6:4257 , piece_id=C2V7ZWJZGPS7KZRU5T6UNMGOF5YWN6JTP6AUZ2E7I2H2ZS3HAAEA, seg_pos={0 0}, upload=527890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:35142->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:35142->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35142->145.79.227.6:4257: use of closed network connection Dec 30 17:45:45.958469029 (602h16m45.829552094s) [noise=true , address=145.79.227.6:4257 , piece_id=C2V7ZWJZGPS7KZRU5T6UNMGOF5YWN6JTP6AUZ2E7I2H2ZS3HAAEA, seg_pos={0 0}, upload=527890]: Long tail cancellation due to optimal threshold Dec 30 17:45:45.958745265 (602h16m45.82982831s) [noise=true , address=145.79.227.3:4151 , piece_id=BLUNO6N5WXNON227NQB2MDJTF5X2OSHOW3EWHG4USXWJ5OMH2BCQ, seg_pos={0 0}, upload=527890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:48350->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:48350->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48350->145.79.227.3:4151: use of closed network connection Dec 30 17:45:45.958755043 (602h16m45.829838089s) [noise=true , address=145.79.227.3:4151 , piece_id=BLUNO6N5WXNON227NQB2MDJTF5X2OSHOW3EWHG4USXWJ5OMH2BCQ, seg_pos={0 0}, upload=527890]: Long tail cancellation due to optimal threshold Dec 30 17:45:45.958775753 (602h16m45.829858808s) [seg_pos={0 0}, upload=527890]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 17:45:45.958880272 (602h16m45.829963317s) [upload=527890]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 17:45:45.958887376 (602h16m45.829970411s) [upload=527890]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 17:45:46.257493423 (602h16m46.128576458s) [upload=527890]: Done uploading segments... Dec 30 17:45:46.257516557 (602h16m46.128599602s) [upload=527890]: Upload finished. err: <nil> Duration:* 5.811991636s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 6.923327872s trace: https://jaeger.dev.storj.io/trace/6ea66b1ed6f4c6e5 FAIL (2025-12-30 17:46:49 UTC) orbiter warning: context deadline exceeded Duration: 6.923356337s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [X...X.......................................................] duration: 5.931945982s trace: https://jaeger.dev.storj.io/trace/6ea66b1ed6f4c6e5 FAIL (2025-12-30 17:46:32 UTC) orbiter warning: context deadline exceeded Duration: 5.931975599s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 27.036052087s trace: https://jaeger.dev.storj.io/trace/5ea8d7d4f61e13a2 FAIL (2025-12-30 17:46:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1X5A7bLqMEQuaMbf9xZ7PDUD4161sz2QqdM9r59y3bsyPBJhaK, Piece ID: HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ) context deadline exceeded; piecestore: (Node ID: 12eLKUg2vTnZMrLaraSPEoGXe3jDVBJYyURdcn4f2gCN5pvvCQ8, Piece ID: KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ) context deadline exceeded; piecestore: (Node ID: 12W4n6uf6azQk1QgeR8Xw7UkpnZm19zdyFKwojCsqZt2GFHrpce, Piece ID: DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ) context deadline exceeded ======LOGS====== 17.114181822 (601h42m16.985264867s) [noise=true , address=145.79.227.44:4208 , piece_id=3OXDCHZSOIFS5VQCI4SXSAQFICDCYHRE5NGFBERR33KOTGSZWLDQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.114641387 (601h42m16.985724432s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.115652532 (601h42m16.986735578s) [noise=true , address=77.90.31.17:4790 , piece_id=5RSRESKGDA2YF4NBRF4ZFABBKWJK4NO6VLEXW2Z6EY3OPXH2WPNA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.115937134 (601h42m16.987020179s) [noise=true , address=64.49.13.155:4923 , piece_id=MUJD7XLM2YRLRCHYVSDPBE6VBYLHCOA7HJQP22ZSET4N574XQMJA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.116814986 (601h42m16.987898031s) [noise=true , address=45.91.116.11:4139 , piece_id=72DPEWZ7ZPZZ53UGYV2VCUKLCK7HCM22CDBRHWVLJGDGO5B5JADQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.117501182 (601h42m16.988584228s) [noise=true , address=77.90.31.14:4665 , piece_id=DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.118263334 (601h42m16.989346379s) [noise=true , address=145.79.227.40:4084 , piece_id=ZADZDRKBIILLAI4IT335CYETOSFOFUUPQIH6HZJCTHKRTZSPDJUQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.118555850 (601h42m16.989638886s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.118765930 (601h42m16.989849036s) [noise=true , address=77.90.31.21:4905 , piece_id=HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.119029533 (601h42m16.990112568s) [noise=true , address=77.90.31.10:4533 , piece_id=QFBBYNQHQL2G7PEREK43FZBRUMKMVNBC2XKTE2NWTVTKN2452LPA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.119417672 (601h42m16.990500707s) [noise=true , address=45.91.116.12:4258 , piece_id=KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.119441588 (601h42m16.990524643s) [noise=true , address=145.79.227.40:4053 , piece_id=JCSIUMWKEAQNQXTNSJTN6VSB3V2SIDRQK67QMK46B4GOUWEMQE2A, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.119763470 (601h42m16.990846505s) [noise=true , address=77.90.31.18:4821 , piece_id=MXZFKXM3JV2HQURDLIU7BKWLAKDZ27O4ER3SGC7QBZV5I3J2EHKA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.120028685 (601h42m16.99111172s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.120086986 (601h42m16.991170031s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.120586507 (601h42m16.991669562s) [noise=true , address=45.91.116.13:4168 , piece_id=AKUE5H3A2UECQOZY3GV3MWZKIHSNKCDOG32DKCNHIRTJROJ74M3A, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.121258768 (601h42m16.992341813s) [noise=true , address=145.79.227.9:4379 , piece_id=QTFXFSGGKYBQSPWJYBBJMQUL3D5EEO3IX4C53ECD7FXMUOGUBC7Q, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:18.568739165 (601h42m18.43982221s) [upload=527370]: Next returned nil segment Dec 30 17:11:18.568751328 (601h42m18.439834363s) [upload=527370]: Waiting for error group managing segments... Dec 30 17:11:18.978424178 (601h42m18.849507213s) [noise=true , address=64.49.13.155:4923 , piece_id=MUJD7XLM2YRLRCHYVSDPBE6VBYLHCOA7HJQP22ZSET4N574XQMJA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:18.986258125 (601h42m18.85734116s) [noise=true , address=145.79.227.9:4379 , piece_id=QTFXFSGGKYBQSPWJYBBJMQUL3D5EEO3IX4C53ECD7FXMUOGUBC7Q, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.274519421 (601h42m19.145602456s) [noise=true , address=77.90.31.14:4665 , piece_id=DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.315775438 (601h42m19.186858473s) [noise=true , address=45.91.116.13:4168 , piece_id=AKUE5H3A2UECQOZY3GV3MWZKIHSNKCDOG32DKCNHIRTJROJ74M3A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.335765371 (601h42m19.206848406s) [noise=true , address=77.90.31.21:4905 , piece_id=HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.354812129 (601h42m19.225895164s) [noise=true , address=77.90.31.10:4533 , piece_id=QFBBYNQHQL2G7PEREK43FZBRUMKMVNBC2XKTE2NWTVTKN2452LPA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.392024917 (601h42m19.263107952s) [noise=true , address=45.91.116.11:4139 , piece_id=72DPEWZ7ZPZZ53UGYV2VCUKLCK7HCM22CDBRHWVLJGDGO5B5JADQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.396084327 (601h42m19.267167372s) [noise=true , address=77.90.31.18:4821 , piece_id=MXZFKXM3JV2HQURDLIU7BKWLAKDZ27O4ER3SGC7QBZV5I3J2EHKA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.862099099 (601h42m19.733182134s) [noise=true , address=77.90.31.17:4790 , piece_id=5RSRESKGDA2YF4NBRF4ZFABBKWJK4NO6VLEXW2Z6EY3OPXH2WPNA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.905412913 (601h42m19.776495948s) [noise=true , address=145.79.227.44:4208 , piece_id=3OXDCHZSOIFS5VQCI4SXSAQFICDCYHRE5NGFBERR33KOTGSZWLDQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.910309688 (601h42m19.781392743s) [noise=true , address=145.79.227.30:4794 , piece_id=EJH6Q3T2BMPVLG6F7CW7N4GYOA3EZFPADL6VPT7WKKG65B2WZVXQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.934233361 (601h42m19.805316397s) [noise=true , address=145.79.227.40:4084 , piece_id=ZADZDRKBIILLAI4IT335CYETOSFOFUUPQIH6HZJCTHKRTZSPDJUQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:20.074011527 (601h42m19.945094572s) [noise=true , address=45.91.116.12:4258 , piece_id=KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:20.106010337 (601h42m19.977093372s) [noise=true , address=145.79.227.40:4053 , piece_id=JCSIUMWKEAQNQXTNSJTN6VSB3V2SIDRQK67QMK46B4GOUWEMQE2A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:20.106080911 (601h42m19.977163946s) [seg_pos={0 0}, upload=527370]: Segment reached optimal threshold of 14 pieces. Dec 30 17:11:20.106274520 (601h42m19.977357565s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Dec 30 17:11:20.106281644 (601h42m19.977364679s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold Dec 30 17:11:20.106351607 (601h42m19.977434642s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF Dec 30 17:11:20.106367767 (601h42m19.977450802s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold Dec 30 17:11:20.106424967 (601h42m19.977508002s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Dec 30 17:11:20.106432451 (601h42m19.977515486s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold Dec 30 17:11:20.106540666 (601h42m19.977623702s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 30 17:11:20.106546207 (601h42m19.977629242s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold Dec 30 17:11:20.106563380 (601h42m19.977646415s) [seg_pos={0 0}, upload=527370]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 17:11:20.106625529 (601h42m19.977708564s) [upload=527370]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 17:11:20.106629837 (601h42m19.977712882s) [upload=527370]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 17:11:20.406465452 (601h42m20.277548507s) [upload=527370]: Done uploading segments... Dec 30 17:11:20.406504206 (601h42m20.277587251s) [upload=527370]: Upload finished. err: <nil> Duration: 27.036079701s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [.X.X....X.XXXXX.XX.X.XXX......XX.XXX...XX..X.....X.XXX....XX] duration: 30.00215876s trace: https://jaeger.dev.storj.io/trace/7499febccbc62e67 FAIL (2025-12-30 17:46:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 123269 (602h16m42.477206304s) [upload=527890]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 17:45:42.876710105 (602h16m42.74779315s) [seg_pos={0 0}, upload=527890]: Begin upload segment... Dec 30 17:45:42.876758677 (602h16m42.747841712s) [seg_pos={0 0}, upload=527890]: Done begin upload segment. Dec 30 17:45:42.876792852 (602h16m42.747875897s) [noise=true , address=45.91.116.12:4228 , piece_id=TEC23262YRTNV2WNAEBS5BZGTDQTMNSZCZG34OOR66HFKB5GQHHA, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.876848779 (602h16m42.747931824s) [noise=true , address=145.79.227.41:4087 , piece_id=JXD32DPHKLSQH36JRJZRPH6SI7ZZAMBL4AM64C6U6KSANSJWLZ3Q, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.877414867 (602h16m42.748497902s) [noise=true , address=145.79.227.3:4151 , piece_id=BLUNO6N5WXNON227NQB2MDJTF5X2OSHOW3EWHG4USXWJ5OMH2BCQ, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.877585782 (602h16m42.748668827s) [noise=true , address=45.91.116.4:4937 , piece_id=VN533XKJKWMBE5CT6N3BRY6UDZS43XCM44DMMR5JNJR5POKG6BJQ, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.877780573 (602h16m42.748863619s) [noise=true , address=145.79.227.12:4765 , piece_id=CTUCAP6LFRR4ZHSTI32NNWIUKCC6DIPWWTCI3Q55OYAUSJ4LQVKQ, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.877983619 (602h16m42.749066664s) [noise=true , address=145.79.227.8:4331 , piece_id=6GH4TYPK3CI3E3DKFRVN3FE6KGOPJ5QAWS44D5SV5F2GKGEEHQQA, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.878158562 (602h16m42.749241597s) [noise=true , address=145.79.227.7:4293 , piece_id=63XKHXW5V53VPRBL4ZE6LNPKFP3E4DX5VDZXQFLMDO2VGJT37M5Q, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.878398479 (602h16m42.749481524s) [noise=true , address=45.91.116.8:4046 , piece_id=A2VZG6A7PGGFNSEVMU2546FVCTLGM6UOYFV5B4SQ6LWECLD6MF2Q, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.878505303 (602h16m42.749588338s) [noise=true , address=145.79.227.51:4410 , piece_id=J7HM7D5JXY3LZBK4RKFOJ4XSNFWHHUNZAN4ZZRMFID2BZURDTXOA, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.878793262 (602h16m42.749876297s) [noise=true , address=145.79.227.39:4013 , piece_id=SAAZXLKDLXEY5QZEQANU7IFSE4E3NTC6N6XZJJ2C44EOW56OSJTA, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.878939831 (602h16m42.750022866s) [noise=true , address=45.91.116.11:4119 , piece_id=WYFKQ22R6IHWPJZ6EVH7IBFTIZCAR4ZRU75EUUBXMER2NIA2YKRA, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.879131355 (602h16m42.7502144s) [noise=true , address=145.79.227.25:4619 , piece_id=VNJVQJ3PVRV4FEPOTSNS3F5JXQ4GUX6VVDZT52GCZNEYNOQMHPMA, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.887768982 (602h16m42.758852017s) [noise=true , address=77.90.31.16:4735 , piece_id=BJGIVHNGAPFCJQH2NYOGUMNLQAA7APG4EVEPQNHIJNP4XHUHKQMQ, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.897314348 (602h16m42.768397383s) [noise=true , address=77.90.31.12:4590 , piece_id=PS4U5VCVVTOAJ4JGVDVOQIIOPS6MH2Y3KY724LKKK5KLUHADMKZQ, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.899145295 (602h16m42.77022833s) [noise=true , address=145.79.227.6:4257 , piece_id=C2V7ZWJZGPS7KZRU5T6UNMGOF5YWN6JTP6AUZ2E7I2H2ZS3HAAEA, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:42.899894471 (602h16m42.770977506s) [noise=true , address=145.79.227.21:4475 , piece_id=RB3FL546I4SVN3FRD7KL6FMYBX7D43WQYEAZUIMMRSJVZTHSDKLQ, seg_pos={0 0}, upload=527890]: Uploading piece... Dec 30 17:45:44.520498726 (602h16m44.391581761s) [upload=527890]: Next returned nil segment Dec 30 17:45:44.520512822 (602h16m44.391595857s) [upload=527890]: Waiting for error group managing segments... Dec 30 17:45:44.862021472 (602h16m44.733104507s) [noise=true , address=145.79.227.7:4293 , piece_id=63XKHXW5V53VPRBL4ZE6LNPKFP3E4DX5VDZXQFLMDO2VGJT37M5Q, seg_pos={0 0}, upload=527890]: Done uploading piece. err: <nil> Dec 30 17:45:44.868822802 (602h16m44.739905848s) [noise=true , address=145.79.227.51:4410 , piece_id=J7HM7D5JXY3LZBK4RKFOJ4XSNFWHHUNZAN4ZZRMFID2BZURDTXOA, seg_pos={0 0}, upload=527890]: Done uploading piece. err: <nil> Dec 30 17:45:44.870439721 (602h16m44.741522756s) [noise=true , address=145.79.227.25:4619 , piece_id=VNJVQJ3PVRV4FEPOTSNS3F5JXQ4GUX6VVDZT52GCZNEYNOQMHPMA, seg_pos={0 0}, upload=527890]: Done uploading piece. err: <nil> Dec 30 17:45:44.871615740 (602h16m44.742698786s) [noise=true , address=145.79.227.39:4013 , piece_id=SAAZXLKDLXEY5QZEQANU7IFSE4E3NTC6N6XZJJ2C44EOW56OSJTA, seg_pos={0 0}, upload=527890]: Done uploading piece. err: <nil> Dec 30 17:45:44.883505831 (602h16m44.754588866s) [noise=true , address=145.79.227.12:4765 , piece_id=CTUCAP6LFRR4ZHSTI32NNWIUKCC6DIPWWTCI3Q55OYAUSJ4LQVKQ, seg_pos={0 0}, upload=527890]: Done uploading piece. err: <nil> Dec 30 17:45:44.889839371 (602h16m44.760922406s) [noise=true , address=145.79.227.8:4331 , piece_id=6GH4TYPK3CI3E3DKFRVN3FE6KGOPJ5QAWS44D5SV5F2GKGEEHQQA, seg_pos={0 0}, upload=527890]: Done uploading piece. err: <nil> Dec 30 17:45:44.950692544 (602h16m44.821775589s) [noise=true , address=145.79.227.41:4087 , piece_id=JXD32DPHKLSQH36JRJZRPH6SI7ZZAMBL4AM64C6U6KSANSJWLZ3Q, seg_pos={0 0}, upload=527890]: Done uploading piece. err: <nil> Dec 30 17:45:44.970768812 (602h16m44.841851847s) [noise=true , address=145.79.227.21:4475 , piece_id=RB3FL546I4SVN3FRD7KL6FMYBX7D43WQYEAZUIMMRSJVZTHSDKLQ, seg_pos={0 0}, upload=527890]: Done uploading piece. err: <nil> Dec 30 17:45:45.153744544 (602h16m45.024827579s) [noise=true , address=77.90.31.16:4735 , piece_id=BJGIVHNGAPFCJQH2NYOGUMNLQAA7APG4EVEPQNHIJNP4XHUHKQMQ, seg_pos={0 0}, upload=527890]: Done uploading piece. err: <nil> Dec 30 17:45:45.155910799 (602h16m45.026993844s) [noise=true , address=77.90.31.12:4590 , piece_id=PS4U5VCVVTOAJ4JGVDVOQIIOPS6MH2Y3KY724LKKK5KLUHADMKZQ, seg_pos={0 0}, upload=527890]: Done uploading piece. err: <nil> Dec 30 17:45:45.190278409 (602h16m45.061361464s) [noise=true , address=45.91.116.11:4119 , piece_id=WYFKQ22R6IHWPJZ6EVH7IBFTIZCAR4ZRU75EUUBXMER2NIA2YKRA, seg_pos={0 0}, upload=527890]: Done uploading piece. err: <nil> Dec 30 17:45:45.285402168 (602h16m45.156485203s) [noise=true , address=45.91.116.12:4228 , piece_id=TEC23262YRTNV2WNAEBS5BZGTDQTMNSZCZG34OOR66HFKB5GQHHA, seg_pos={0 0}, upload=527890]: Done uploading piece. err: <nil> Dec 30 17:45:45.958179638 (602h16m45.829262683s) [noise=true , address=45.91.116.4:4937 , piece_id=VN533XKJKWMBE5CT6N3BRY6UDZS43XCM44DMMR5JNJR5POKG6BJQ, seg_pos={0 0}, upload=527890]: Done uploading piece. err: <nil> Dec 30 17:45:45.958204887 (602h16m45.829287952s) [seg_pos={0 0}, upload=527890]: Segment reached optimal threshold of 13 pieces. Dec 30 17:45:45.958425907 (602h16m45.829508962s) [noise=true , address=45.91.116.8:4046 , piece_id=A2VZG6A7PGGFNSEVMU2546FVCTLGM6UOYFV5B4SQ6LWECLD6MF2Q, seg_pos={0 0}, upload=527890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H): protocol: write tcp 188.166.225.170:51886->45.91.116.8:4046: use of closed network connection; write tcp 188.166.225.170:51886->45.91.116.8:4046: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51886->45.91.116.8:4046: use of closed network connection Dec 30 17:45:45.958435716 (602h16m45.829518771s) [noise=true , address=45.91.116.8:4046 , piece_id=A2VZG6A7PGGFNSEVMU2546FVCTLGM6UOYFV5B4SQ6LWECLD6MF2Q, seg_pos={0 0}, upload=527890]: Long tail cancellation due to optimal threshold Dec 30 17:45:45.958459822 (602h16m45.829542877s) [noise=true , address=145.79.227.6:4257 , piece_id=C2V7ZWJZGPS7KZRU5T6UNMGOF5YWN6JTP6AUZ2E7I2H2ZS3HAAEA, seg_pos={0 0}, upload=527890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:35142->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:35142->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35142->145.79.227.6:4257: use of closed network connection Dec 30 17:45:45.958469029 (602h16m45.829552094s) [noise=true , address=145.79.227.6:4257 , piece_id=C2V7ZWJZGPS7KZRU5T6UNMGOF5YWN6JTP6AUZ2E7I2H2ZS3HAAEA, seg_pos={0 0}, upload=527890]: Long tail cancellation due to optimal threshold Dec 30 17:45:45.958745265 (602h16m45.82982831s) [noise=true , address=145.79.227.3:4151 , piece_id=BLUNO6N5WXNON227NQB2MDJTF5X2OSHOW3EWHG4USXWJ5OMH2BCQ, seg_pos={0 0}, upload=527890]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB): protocol: write tcp 188.166.225.170:48350->145.79.227.3:4151: use of closed network connection; write tcp 188.166.225.170:48350->145.79.227.3:4151: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48350->145.79.227.3:4151: use of closed network connection Dec 30 17:45:45.958755043 (602h16m45.829838089s) [noise=true , address=145.79.227.3:4151 , piece_id=BLUNO6N5WXNON227NQB2MDJTF5X2OSHOW3EWHG4USXWJ5OMH2BCQ, seg_pos={0 0}, upload=527890]: Long tail cancellation due to optimal threshold Dec 30 17:45:45.958775753 (602h16m45.829858808s) [seg_pos={0 0}, upload=527890]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 17:45:45.958880272 (602h16m45.829963317s) [upload=527890]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 17:45:45.958887376 (602h16m45.829970411s) [upload=527890]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 17:45:46.257493423 (602h16m46.128576458s) [upload=527890]: Done uploading segments... Dec 30 17:45:46.257516557 (602h16m46.128599602s) [upload=527890]: Upload finished. err: <nil> Duration:* 30.002262607s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [....X..X....X.XX..X..X......X.........X..XXX................] duration: 5.001168496s trace: https://jaeger.dev.storj.io/trace/56692ebd5ab65c7c FAIL (2025-12-30 17:46:07 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00126474s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......X...X..X............................XX...............] duration: 6.173535995s trace: https://jaeger.dev.storj.io/trace/4173ea2fcd946b89 FAIL (2025-12-30 17:45:46 UTC) orbiter warning: context deadline exceeded Duration: 6.173569308s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [X.X.X....X.XXXXX.XX.X.XXX......XX.XXX...XX..X.....X.XXX....X] duration: 30.003727389s trace: https://jaeger.dev.storj.io/trace/33fbc7783ac70881 FAIL (2025-12-30 17:45:41 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 30 17:45:12.345127087 (602h16m12.216210122s) [upload=527882]: Starting upload Dec 30 17:45:12.345197601 (602h16m12.216280666s) [upload=527882]: Done starting upload Dec 30 17:45:12.345337587 (602h16m12.216420632s) [upload=527882]: Uploading segments... Dec 30 17:45:12.345482984 (602h16m12.216566029s) [upload=527882]: Got next segment. Inline: false Dec 30 17:45:12.345580360 (602h16m12.216663405s) [upload=527882]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 17:45:12.345588936 (602h16m12.216671981s) [upload=527882]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 17:45:12.589709435 (602h16m12.46079249s) [seg_pos={0 0}, upload=527882]: Begin upload segment... Dec 30 17:45:12.589766483 (602h16m12.460849528s) [seg_pos={0 0}, upload=527882]: Done begin upload segment. Dec 30 17:45:12.589834462 (602h16m12.460917507s) [noise=true , address=145.79.227.6:4259 , piece_id=BRLYMMQL2BKA36TJJMVRWQ3YXSUV5UOOS544SO7TM4HICVAWSCMQ, seg_pos={0 0}, upload=527882]: Uploading piece... Dec 30 17:45:12.589963698 (602h16m12.461046733s) [noise=true , address=45.91.116.13:4155 , piece_id=FGAYHOH5XFSS6EHMCZNRXT7HQ7NQ7FL5TDBBYW7M4HRNU4ME5WFA, seg_pos={0 0}, upload=527882]: Uploading piece... Dec 30 17:45:12.590545727 (602h16m12.461628772s) [noise=true , address=145.79.227.14:4839 , piece_id=4XTBHU53SKPC6YXTXAX4BRO5CFDYYKZS6BPCWZV52JCZYKDM4D4Q, seg_pos={0 0}, upload=527882]: Uploading piece... Dec 30 17:45:12.591093871 (602h16m12.462176916s) [noise=true , address=145.79.227.36:4517 , piece_id=ZTQUQPJ565VJWY34GXAM7K6SPAFBZBSJNM4CY74DCQMUJN3MGB4Q, seg_pos={0 0}, upload=527882]: Uploading piece... Dec 30 17:45:12.591705004 (602h16m12.462788039s) [noise=true , address=45.91.116.6:4976 , piece_id=JD5VDYJUPPIJJ64ESHGP7QTZIFCI5GNBOSCFW4U5VQBKU54AKMYA, seg_pos={0 0}, upload=527882]: Uploading piece... Dec 30 17:45:12.592111779 (602h16m12.463194814s) [noise=true , address=145.79.227.19:4077 , piece_id=TTGCY7NWEQ3XHMJ237UDEXFTH4I5CJSFB3AIPHZVX2RXTEBLRGCQ, seg_pos={0 0}, upload=527882]: Uploading piece... Dec 30 17:45:12.592546336 (602h16m12.463629371s) [noise=true , address=77.90.31.21:4880 , piece_id=HWRKQT7RPHJIUWYANQGAYHETNPPQOX7JIMCCXMTO772RM4ZOPBJQ, seg_pos={0 0}, upload=527882]: Uploading piece... Dec 30 17:45:12.592956768 (602h16m12.464039803s) [noise=true , address=145.79.227.13:4803 , piece_id=5LF26MK3XDUAZNHGI6MKPCXFOBVO4ZQ6LOH3EIPTGFRDBXJEY3DA, seg_pos={0 0}, upload=527882]: Uploading piece... Dec 30 17:45:12.593464525 (602h16m12.46454757s) [noise=true , address=77.90.31.13:4627 , piece_id=QWI44PWSKRQU2HDADAJAASLLNRP4FQWNLIEI6PU7RI24ZNCNFOJQ, seg_pos={0 0}, upload=527882]: Uploading piece... Dec 30 17:45:12.593562431 (602h16m12.464645556s) [noise=true , address=145.79.227.8:4329 , piece_id=IDXBPLT7IRLNBUBGW5OA46ZQMYKJRR4YFACHOJGNZTNGJYBV3VPQ, seg_pos={0 0}, upload=527882]: Uploading piece... Dec 30 17:45:12.594098873 (602h16m12.465181908s) [noise=true , address=145.79.227.15:4875 , piece_id=BBGJJYNQTGL4IOTARFAX7EMF7DQMLIV4L6SVJ4RUZAKOJ4RNIB7A, seg_pos={0 0}, upload=527882]: Uploading piece... Dec 30 17:45:12.594405046 (602h16m12.465488081s) [noise=true , address=145.79.227.50:4375 , piece_id=SVGLJLNBNXIK6DVZTEHCAONK35XLSMTZDWRWTMCJMTQTSNZPNVPA, seg_pos={0 0}, upload=527882]: Uploading piece... Dec 30 17:45:12.594727329 (602h16m12.465810374s) [noise=true , address=77.90.31.19:4841 , piece_id=E66QRXKSIOMFT542QCNU4RCBY5OBMKOHWEEUGLWZGTWGK75IE7IA, seg_pos={0 0}, upload=527882]: Uploading piece... Dec 30 17:45:12.595030958 (602h16m12.466113993s) [noise=true , address=145.79.227.37:4566 , piece_id=EP5GY63SAOLXLKJC5EN5TFV4OOXAVVDHUOPDQICEZDQV5TDQAW2A, seg_pos={0 0}, upload=527882]: Uploading piece... Dec 30 17:45:12.595425248 (602h16m12.466508283s) [noise=true , address=145.79.227.2:4115 , piece_id=EBPBKG6CGQOJK3RN7URJ5D2H2WF3IKYVQXGZGJZJW4RRXAZ4OOHQ, seg_pos={0 0}, upload=527882]: Uploading piece... Dec 30 17:45:12.595731842 (602h16m12.466814877s) [noise=true , address=45.91.116.10:4081 , piece_id=CVQDPS4JXX22RDWCXHJ6FF2JUJ37ET3QDD5CHLEBNIGZTVOSEGAA, seg_pos={0 0}, upload=527882]: Uploading piece... Dec 30 17:45:14.285121254 (602h16m14.156204299s) [upload=527882]: Next returned nil segment Dec 30 17:45:14.285133978 (602h16m14.156217023s) [upload=527882]: Waiting for error group managing segments... Dec 30 17:45:14.575844490 (602h16m14.446927526s) [noise=true , address=145.79.227.50:4375 , piece_id=SVGLJLNBNXIK6DVZTEHCAONK35XLSMTZDWRWTMCJMTQTSNZPNVPA, seg_pos={0 0}, upload=527882]: Done uploading piece. err: <nil> Dec 30 17:45:14.583138871 (602h16m14.454221906s) [noise=true , address=145.79.227.6:4259 , piece_id=BRLYMMQL2BKA36TJJMVRWQ3YXSUV5UOOS544SO7TM4HICVAWSCMQ, seg_pos={0 0}, upload=527882]: Done uploading piece. err: <nil> Dec 30 17:45:14.585169287 (602h16m14.456252332s) [noise=true , address=145.79.227.15:4875 , piece_id=BBGJJYNQTGL4IOTARFAX7EMF7DQMLIV4L6SVJ4RUZAKOJ4RNIB7A, seg_pos={0 0}, upload=527882]: Done uploading piece. err: <nil> Dec 30 17:45:14.587381939 (602h16m14.458464984s) [noise=true , address=145.79.227.36:4517 , piece_id=ZTQUQPJ565VJWY34GXAM7K6SPAFBZBSJNM4CY74DCQMUJN3MGB4Q, seg_pos={0 0}, upload=527882]: Done uploading piece. err: <nil> Dec 30 17:45:14.595074187 (602h16m14.466157222s) [noise=true , address=145.79.227.8:4329 , piece_id=IDXBPLT7IRLNBUBGW5OA46ZQMYKJRR4YFACHOJGNZTNGJYBV3VPQ, seg_pos={0 0}, upload=527882]: Done uploading piece. err: <nil> Dec 30 17:45:14.597450703 (602h16m14.468533768s) [noise=true , address=145.79.227.13:4803 , piece_id=5LF26MK3XDUAZNHGI6MKPCXFOBVO4ZQ6LOH3EIPTGFRDBXJEY3DA, seg_pos={0 0}, upload=527882]: Done uploading piece. err: <nil> Dec 30 17:45:14.602199625 (602h16m14.47328266s) [noise=true , address=145.79.227.2:4115 , piece_id=EBPBKG6CGQOJK3RN7URJ5D2H2WF3IKYVQXGZGJZJW4RRXAZ4OOHQ, seg_pos={0 0}, upload=527882]: Done uploading piece. err: <nil> Dec 30 17:45:14.610142379 (602h16m14.481225424s) [noise=true , address=145.79.227.37:4566 , piece_id=EP5GY63SAOLXLKJC5EN5TFV4OOXAVVDHUOPDQICEZDQV5TDQAW2A, seg_pos={0 0}, upload=527882]: Done uploading piece. err: <nil> Dec 30 17:45:14.613449305 (602h16m14.48453235s) [noise=true , address=145.79.227.14:4839 , piece_id=4XTBHU53SKPC6YXTXAX4BRO5CFDYYKZS6BPCWZV52JCZYKDM4D4Q, seg_pos={0 0}, upload=527882]: Done uploading piece. err: <nil> Dec 30 17:45:14.614162273 (602h16m14.485245308s) [noise=true , address=145.79.227.19:4077 , piece_id=TTGCY7NWEQ3XHMJ237UDEXFTH4I5CJSFB3AIPHZVX2RXTEBLRGCQ, seg_pos={0 0}, upload=527882]: Done uploading piece. err: <nil> Dec 30 17:45:14.756167851 (602h16m14.627250886s) [noise=true , address=77.90.31.13:4627 , piece_id=QWI44PWSKRQU2HDADAJAASLLNRP4FQWNLIEI6PU7RI24ZNCNFOJQ, seg_pos={0 0}, upload=527882]: Done uploading piece. err: <nil> Dec 30 17:45:14.785930834 (602h16m14.657013879s) [noise=true , address=77.90.31.21:4880 , piece_id=HWRKQT7RPHJIUWYANQGAYHETNPPQOX7JIMCCXMTO772RM4ZOPBJQ, seg_pos={0 0}, upload=527882]: Done uploading piece. err: <nil> Dec 30 17:45:14.986015463 (602h16m14.857098498s) [noise=true , address=45.91.116.6:4976 , piece_id=JD5VDYJUPPIJJ64ESHGP7QTZIFCI5GNBOSCFW4U5VQBKU54AKMYA, seg_pos={0 0}, upload=527882]: Done uploading piece. err: <nil> Dec 30 17:45:14.986037024 (602h16m14.857120059s) [seg_pos={0 0}, upload=527882]: Segment reached optimal threshold of 13 pieces. Dec 30 17:45:14.986238358 (602h16m14.857321393s) [noise=true , address=45.91.116.13:4155 , piece_id=FGAYHOH5XFSS6EHMCZNRXT7HQ7NQ7FL5TDBBYW7M4HRNU4ME5WFA, seg_pos={0 0}, upload=527882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 30 17:45:14.986245191 (602h16m14.857328236s) [noise=true , address=45.91.116.13:4155 , piece_id=FGAYHOH5XFSS6EHMCZNRXT7HQ7NQ7FL5TDBBYW7M4HRNU4ME5WFA, seg_pos={0 0}, upload=527882]: Long tail cancellation due to optimal threshold Dec 30 17:45:14.986438448 (602h16m14.857521483s) [noise=true , address=45.91.116.10:4081 , piece_id=CVQDPS4JXX22RDWCXHJ6FF2JUJ37ET3QDD5CHLEBNIGZTVOSEGAA, seg_pos={0 0}, upload=527882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Dec 30 17:45:14.986444661 (602h16m14.857527696s) [noise=true , address=45.91.116.10:4081 , piece_id=CVQDPS4JXX22RDWCXHJ6FF2JUJ37ET3QDD5CHLEBNIGZTVOSEGAA, seg_pos={0 0}, upload=527882]: Long tail cancellation due to optimal threshold Dec 30 17:45:14.986617460 (602h16m14.857700515s) [noise=true , address=77.90.31.19:4841 , piece_id=E66QRXKSIOMFT542QCNU4RCBY5OBMKOHWEEUGLWZGTWGK75IE7IA, seg_pos={0 0}, upload=527882]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: write tcp 188.166.225.170:51842->77.90.31.19:4841: use of closed network connection; write tcp 188.166.225.170:51842->77.90.31.19:4841: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51842->77.90.31.19:4841: use of closed network connection Dec 30 17:45:14.986639131 (602h16m14.857722196s) [noise=true , address=77.90.31.19:4841 , piece_id=E66QRXKSIOMFT542QCNU4RCBY5OBMKOHWEEUGLWZGTWGK75IE7IA, seg_pos={0 0}, upload=527882]: Long tail cancellation due to optimal threshold Dec 30 17:45:14.986695188 (602h16m14.857778233s) [seg_pos={0 0}, upload=527882]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 17:45:14.986875691 (602h16m14.857958746s) [upload=527882]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 17:45:14.986884768 (602h16m14.857967813s) [upload=527882]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 17:45:15.294940039 (602h16m15.166023074s) [upload=527882]: Done uploading segments... Dec 30 17:45:15.294992659 (602h16m15.166075694s) [upload=527882]: Upload finished. err: <nil> Duration: 30.003742408s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.X...X......................................................] duration: 6.126184047s trace: https://jaeger.dev.storj.io/trace/6ea66b1ed6f4c6e5 FAIL (2025-12-30 17:45:30 UTC) orbiter warning: context deadline exceeded Duration: 6.126211921s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 37.928124416s trace: https://jaeger.dev.storj.io/trace/5ea8d7d4f61e13a2 FAIL (2025-12-30 17:45:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1X5A7bLqMEQuaMbf9xZ7PDUD4161sz2QqdM9r59y3bsyPBJhaK, Piece ID: HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ) context deadline exceeded; piecestore: (Node ID: 12eLKUg2vTnZMrLaraSPEoGXe3jDVBJYyURdcn4f2gCN5pvvCQ8, Piece ID: KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ) context deadline exceeded; piecestore: (Node ID: 12W4n6uf6azQk1QgeR8Xw7UkpnZm19zdyFKwojCsqZt2GFHrpce, Piece ID: DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ) context deadline exceeded ======LOGS====== 17.114181822 (601h42m16.985264867s) [noise=true , address=145.79.227.44:4208 , piece_id=3OXDCHZSOIFS5VQCI4SXSAQFICDCYHRE5NGFBERR33KOTGSZWLDQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.114641387 (601h42m16.985724432s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.115652532 (601h42m16.986735578s) [noise=true , address=77.90.31.17:4790 , piece_id=5RSRESKGDA2YF4NBRF4ZFABBKWJK4NO6VLEXW2Z6EY3OPXH2WPNA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.115937134 (601h42m16.987020179s) [noise=true , address=64.49.13.155:4923 , piece_id=MUJD7XLM2YRLRCHYVSDPBE6VBYLHCOA7HJQP22ZSET4N574XQMJA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.116814986 (601h42m16.987898031s) [noise=true , address=45.91.116.11:4139 , piece_id=72DPEWZ7ZPZZ53UGYV2VCUKLCK7HCM22CDBRHWVLJGDGO5B5JADQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.117501182 (601h42m16.988584228s) [noise=true , address=77.90.31.14:4665 , piece_id=DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.118263334 (601h42m16.989346379s) [noise=true , address=145.79.227.40:4084 , piece_id=ZADZDRKBIILLAI4IT335CYETOSFOFUUPQIH6HZJCTHKRTZSPDJUQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.118555850 (601h42m16.989638886s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.118765930 (601h42m16.989849036s) [noise=true , address=77.90.31.21:4905 , piece_id=HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.119029533 (601h42m16.990112568s) [noise=true , address=77.90.31.10:4533 , piece_id=QFBBYNQHQL2G7PEREK43FZBRUMKMVNBC2XKTE2NWTVTKN2452LPA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.119417672 (601h42m16.990500707s) [noise=true , address=45.91.116.12:4258 , piece_id=KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.119441588 (601h42m16.990524643s) [noise=true , address=145.79.227.40:4053 , piece_id=JCSIUMWKEAQNQXTNSJTN6VSB3V2SIDRQK67QMK46B4GOUWEMQE2A, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.119763470 (601h42m16.990846505s) [noise=true , address=77.90.31.18:4821 , piece_id=MXZFKXM3JV2HQURDLIU7BKWLAKDZ27O4ER3SGC7QBZV5I3J2EHKA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.120028685 (601h42m16.99111172s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.120086986 (601h42m16.991170031s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.120586507 (601h42m16.991669562s) [noise=true , address=45.91.116.13:4168 , piece_id=AKUE5H3A2UECQOZY3GV3MWZKIHSNKCDOG32DKCNHIRTJROJ74M3A, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.121258768 (601h42m16.992341813s) [noise=true , address=145.79.227.9:4379 , piece_id=QTFXFSGGKYBQSPWJYBBJMQUL3D5EEO3IX4C53ECD7FXMUOGUBC7Q, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:18.568739165 (601h42m18.43982221s) [upload=527370]: Next returned nil segment Dec 30 17:11:18.568751328 (601h42m18.439834363s) [upload=527370]: Waiting for error group managing segments... Dec 30 17:11:18.978424178 (601h42m18.849507213s) [noise=true , address=64.49.13.155:4923 , piece_id=MUJD7XLM2YRLRCHYVSDPBE6VBYLHCOA7HJQP22ZSET4N574XQMJA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:18.986258125 (601h42m18.85734116s) [noise=true , address=145.79.227.9:4379 , piece_id=QTFXFSGGKYBQSPWJYBBJMQUL3D5EEO3IX4C53ECD7FXMUOGUBC7Q, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.274519421 (601h42m19.145602456s) [noise=true , address=77.90.31.14:4665 , piece_id=DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.315775438 (601h42m19.186858473s) [noise=true , address=45.91.116.13:4168 , piece_id=AKUE5H3A2UECQOZY3GV3MWZKIHSNKCDOG32DKCNHIRTJROJ74M3A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.335765371 (601h42m19.206848406s) [noise=true , address=77.90.31.21:4905 , piece_id=HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.354812129 (601h42m19.225895164s) [noise=true , address=77.90.31.10:4533 , piece_id=QFBBYNQHQL2G7PEREK43FZBRUMKMVNBC2XKTE2NWTVTKN2452LPA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.392024917 (601h42m19.263107952s) [noise=true , address=45.91.116.11:4139 , piece_id=72DPEWZ7ZPZZ53UGYV2VCUKLCK7HCM22CDBRHWVLJGDGO5B5JADQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.396084327 (601h42m19.267167372s) [noise=true , address=77.90.31.18:4821 , piece_id=MXZFKXM3JV2HQURDLIU7BKWLAKDZ27O4ER3SGC7QBZV5I3J2EHKA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.862099099 (601h42m19.733182134s) [noise=true , address=77.90.31.17:4790 , piece_id=5RSRESKGDA2YF4NBRF4ZFABBKWJK4NO6VLEXW2Z6EY3OPXH2WPNA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.905412913 (601h42m19.776495948s) [noise=true , address=145.79.227.44:4208 , piece_id=3OXDCHZSOIFS5VQCI4SXSAQFICDCYHRE5NGFBERR33KOTGSZWLDQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.910309688 (601h42m19.781392743s) [noise=true , address=145.79.227.30:4794 , piece_id=EJH6Q3T2BMPVLG6F7CW7N4GYOA3EZFPADL6VPT7WKKG65B2WZVXQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.934233361 (601h42m19.805316397s) [noise=true , address=145.79.227.40:4084 , piece_id=ZADZDRKBIILLAI4IT335CYETOSFOFUUPQIH6HZJCTHKRTZSPDJUQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:20.074011527 (601h42m19.945094572s) [noise=true , address=45.91.116.12:4258 , piece_id=KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:20.106010337 (601h42m19.977093372s) [noise=true , address=145.79.227.40:4053 , piece_id=JCSIUMWKEAQNQXTNSJTN6VSB3V2SIDRQK67QMK46B4GOUWEMQE2A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:20.106080911 (601h42m19.977163946s) [seg_pos={0 0}, upload=527370]: Segment reached optimal threshold of 14 pieces. Dec 30 17:11:20.106274520 (601h42m19.977357565s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Dec 30 17:11:20.106281644 (601h42m19.977364679s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold Dec 30 17:11:20.106351607 (601h42m19.977434642s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF Dec 30 17:11:20.106367767 (601h42m19.977450802s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold Dec 30 17:11:20.106424967 (601h42m19.977508002s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Dec 30 17:11:20.106432451 (601h42m19.977515486s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold Dec 30 17:11:20.106540666 (601h42m19.977623702s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 30 17:11:20.106546207 (601h42m19.977629242s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold Dec 30 17:11:20.106563380 (601h42m19.977646415s) [seg_pos={0 0}, upload=527370]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 17:11:20.106625529 (601h42m19.977708564s) [upload=527370]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 17:11:20.106629837 (601h42m19.977712882s) [upload=527370]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 17:11:20.406465452 (601h42m20.277548507s) [upload=527370]: Done uploading segments... Dec 30 17:11:20.406504206 (601h42m20.277587251s) [upload=527370]: Upload finished. err: <nil> Duration: 37.928156697s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [.....X..X....X.XX..X..X......X.........X..XXX...............] duration: 3.636429286s trace: https://jaeger.dev.storj.io/trace/56692ebd5ab65c7c FAIL (2025-12-30 17:45:12 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.636459353s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [........X...X..X............................XX..............] duration: 6.128823931s trace: https://jaeger.dev.storj.io/trace/4173ea2fcd946b89 FAIL (2025-12-30 17:44:36 UTC) orbiter warning: context deadline exceeded Duration: 6.128857194s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [.X.X.X....X.XXXXX.XX.X.XXX......XX.XXX...XX..X.....X.XXX....] duration: 6.35373949s trace: https://jaeger.dev.storj.io/trace/42312365b09db431 FAIL (2025-12-30 17:44:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: 3U772XXF3AB2CZCRKYHSAXLF3B4VW7W2ISZHBUAS3BBSUQF6IDCQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: KF5PY3KZ42T6CFJXFDOOC344DIL6SZANTQ54RAS4RZA6PQMYBGEQ) context deadline exceeded ======LOGS====== g segments... Dec 30 17:41:29.742890691 (602h12m29.613973726s) [upload=527821]: Got next segment. Inline: false Dec 30 17:41:29.742941117 (602h12m29.614024153s) [upload=527821]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 17:41:29.742949253 (602h12m29.614032288s) [upload=527821]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 17:41:29.994240285 (602h12m29.86532332s) [seg_pos={0 0}, upload=527821]: Begin upload segment... Dec 30 17:41:29.994289349 (602h12m29.865372384s) [seg_pos={0 0}, upload=527821]: Done begin upload segment. Dec 30 17:41:29.994316519 (602h12m29.865399555s) [noise=true , address=145.79.227.45:4231 , piece_id=AJR4RUB6DWJRXPN3U5GVN556R4QFWDEJCTUAP6A4FYHPCV67OCSQ, seg_pos={0 0}, upload=527821]: Uploading piece... Dec 30 17:41:29.994414386 (602h12m29.865497421s) [noise=true , address=145.79.227.46:4603 , piece_id=DBKR4JT3SVCLF5RDVFC4XNIYOY7SW5KPLLZSHJ5E5CMHZMJ4VQCQ, seg_pos={0 0}, upload=527821]: Uploading piece... Dec 30 17:41:29.994888609 (602h12m29.865971644s) [noise=true , address=45.91.116.6:4975 , piece_id=KF5PY3KZ42T6CFJXFDOOC344DIL6SZANTQ54RAS4RZA6PQMYBGEQ, seg_pos={0 0}, upload=527821]: Uploading piece... Dec 30 17:41:29.995193370 (602h12m29.866276405s) [noise=true , address=145.79.227.39:4014 , piece_id=BNWQMRUHNUN5P3ICNNOQWFIY4UYLT2A73BVBPPJS42AUJDHJAHGQ, seg_pos={0 0}, upload=527821]: Uploading piece... Dec 30 17:41:29.995701447 (602h12m29.866784482s) [noise=true , address=145.79.227.52:4446 , piece_id=DT3MQLERNJLSKDDUI5XXI33RWHDFMRQ4Q74VFY5CBKLABSBAQXYA, seg_pos={0 0}, upload=527821]: Uploading piece... Dec 30 17:41:29.996093293 (602h12m29.867176328s) [noise=true , address=45.91.116.13:4153 , piece_id=Q2XJ6SJVS7O7T6JRWIDLDQTADF7IM3JFOBRGYTDAYCPDH3IYIEOQ, seg_pos={0 0}, upload=527821]: Uploading piece... Dec 30 17:41:29.999308716 (602h12m29.870391751s) [noise=true , address=145.79.227.32:4834 , piece_id=IJ36JBCLTCF3OYU73E24VW7IOMYYD6YZMK6CRYFFNUDZMNA3Z5KA, seg_pos={0 0}, upload=527821]: Uploading piece... Dec 30 17:41:30.000590808 (602h12m29.871673863s) [noise=true , address=145.79.227.17:4970 , piece_id=FN2HC3KAZNAYHHGE3QDSHT6RDEP6YRRUAPUDLEHFPJKCZ2LT4FOA, seg_pos={0 0}, upload=527821]: Uploading piece... Dec 30 17:41:30.002162009 (602h12m29.873245074s) [noise=true , address=145.79.227.27:4691 , piece_id=CYIOOBJCMRFBL6ZNRXMB7S7P5XDWGUHVPHXATEJ7BVGLFGDN3PKQ, seg_pos={0 0}, upload=527821]: Uploading piece... Dec 30 17:41:30.003956816 (602h12m29.875039871s) [noise=true , address=145.79.227.41:4085 , piece_id=ZY5QJKXK33T6UK3AEYTVX2KQFXXUUVVK5BH73DE6TNKKDJVJGNGQ, seg_pos={0 0}, upload=527821]: Uploading piece... Dec 30 17:41:30.005277872 (602h12m29.876360917s) [noise=true , address=145.79.227.43:4158 , piece_id=GHXYRFMQKKQYF3TTT44BNM7MXD44FEKLLA5Q55SRD2AZ3YXFBJPQ, seg_pos={0 0}, upload=527821]: Uploading piece... Dec 30 17:41:30.006712503 (602h12m29.877795539s) [noise=true , address=77.90.31.10:4518 , piece_id=WUFRRFHXLNKOWWUPN3GSTOL7SPDJLNCAF6H5N3KT73UTTTHETXPQ, seg_pos={0 0}, upload=527821]: Uploading piece... Dec 30 17:41:30.007057100 (602h12m29.878140135s) [noise=true , address=145.79.227.47:4266 , piece_id=ZCM557E5SSQPU3Q3RVAOHU4ZXIAZMPPKARICBO7WW6SNIY3B5TSA, seg_pos={0 0}, upload=527821]: Uploading piece... Dec 30 17:41:30.008176651 (602h12m29.879259696s) [noise=true , address=145.79.227.35:4481 , piece_id=UPYL65RVDH5Z7RJOOVB7474ZHLK3665T2DPUXIHC2H67CK3Q56EA, seg_pos={0 0}, upload=527821]: Uploading piece... Dec 30 17:41:30.008429382 (602h12m29.879512427s) [noise=true , address=77.90.31.14:4664 , piece_id=3U772XXF3AB2CZCRKYHSAXLF3B4VW7W2ISZHBUAS3BBSUQF6IDCQ, seg_pos={0 0}, upload=527821]: Uploading piece... Dec 30 17:41:30.009714610 (602h12m29.880797645s) [noise=true , address=77.90.31.16:4736 , piece_id=T34VZ7QZXCZVXDTLGIVJT3P7ROWJUZKVLA2SWIVD5ZVA7NI5KC5A, seg_pos={0 0}, upload=527821]: Uploading piece... Dec 30 17:41:31.711933153 (602h12m31.583016198s) [upload=527821]: Next returned nil segment Dec 30 17:41:31.711961837 (602h12m31.583044882s) [upload=527821]: Waiting for error group managing segments... Dec 30 17:41:31.999912104 (602h12m31.870995149s) [noise=true , address=145.79.227.39:4014 , piece_id=BNWQMRUHNUN5P3ICNNOQWFIY4UYLT2A73BVBPPJS42AUJDHJAHGQ, seg_pos={0 0}, upload=527821]: Done uploading piece. err: <nil> Dec 30 17:41:32.007187788 (602h12m31.878270833s) [noise=true , address=145.79.227.52:4446 , piece_id=DT3MQLERNJLSKDDUI5XXI33RWHDFMRQ4Q74VFY5CBKLABSBAQXYA, seg_pos={0 0}, upload=527821]: Done uploading piece. err: <nil> Dec 30 17:41:32.009116290 (602h12m31.880199335s) [noise=true , address=145.79.227.45:4231 , piece_id=AJR4RUB6DWJRXPN3U5GVN556R4QFWDEJCTUAP6A4FYHPCV67OCSQ, seg_pos={0 0}, upload=527821]: Done uploading piece. err: <nil> Dec 30 17:41:32.011104976 (602h12m31.882188011s) [noise=true , address=145.79.227.32:4834 , piece_id=IJ36JBCLTCF3OYU73E24VW7IOMYYD6YZMK6CRYFFNUDZMNA3Z5KA, seg_pos={0 0}, upload=527821]: Done uploading piece. err: <nil> Dec 30 17:41:32.016382305 (602h12m31.88746534s) [noise=true , address=145.79.227.47:4266 , piece_id=ZCM557E5SSQPU3Q3RVAOHU4ZXIAZMPPKARICBO7WW6SNIY3B5TSA, seg_pos={0 0}, upload=527821]: Done uploading piece. err: <nil> Dec 30 17:41:32.022409591 (602h12m31.893492636s) [noise=true , address=145.79.227.41:4085 , piece_id=ZY5QJKXK33T6UK3AEYTVX2KQFXXUUVVK5BH73DE6TNKKDJVJGNGQ, seg_pos={0 0}, upload=527821]: Done uploading piece. err: <nil> Dec 30 17:41:32.082258722 (602h12m31.953341757s) [noise=true , address=145.79.227.46:4603 , piece_id=DBKR4JT3SVCLF5RDVFC4XNIYOY7SW5KPLLZSHJ5E5CMHZMJ4VQCQ, seg_pos={0 0}, upload=527821]: Done uploading piece. err: <nil> Dec 30 17:41:32.093969341 (602h12m31.965052386s) [noise=true , address=145.79.227.27:4691 , piece_id=CYIOOBJCMRFBL6ZNRXMB7S7P5XDWGUHVPHXATEJ7BVGLFGDN3PKQ, seg_pos={0 0}, upload=527821]: Done uploading piece. err: <nil> Dec 30 17:41:32.106393409 (602h12m31.977476444s) [noise=true , address=145.79.227.17:4970 , piece_id=FN2HC3KAZNAYHHGE3QDSHT6RDEP6YRRUAPUDLEHFPJKCZ2LT4FOA, seg_pos={0 0}, upload=527821]: Done uploading piece. err: <nil> Dec 30 17:41:32.281425142 (602h12m32.152508177s) [noise=true , address=77.90.31.14:4664 , piece_id=3U772XXF3AB2CZCRKYHSAXLF3B4VW7W2ISZHBUAS3BBSUQF6IDCQ, seg_pos={0 0}, upload=527821]: Done uploading piece. err: <nil> Dec 30 17:41:32.327920055 (602h12m32.1990031s) [noise=true , address=45.91.116.6:4975 , piece_id=KF5PY3KZ42T6CFJXFDOOC344DIL6SZANTQ54RAS4RZA6PQMYBGEQ, seg_pos={0 0}, upload=527821]: Done uploading piece. err: <nil> Dec 30 17:41:32.466119095 (602h12m32.33720214s) [noise=true , address=45.91.116.13:4153 , piece_id=Q2XJ6SJVS7O7T6JRWIDLDQTADF7IM3JFOBRGYTDAYCPDH3IYIEOQ, seg_pos={0 0}, upload=527821]: Done uploading piece. err: <nil> Dec 30 17:41:32.687451991 (602h12m32.558535037s) [noise=true , address=145.79.227.43:4158 , piece_id=GHXYRFMQKKQYF3TTT44BNM7MXD44FEKLLA5Q55SRD2AZ3YXFBJPQ, seg_pos={0 0}, upload=527821]: Done uploading piece. err: <nil> Dec 30 17:41:32.687477170 (602h12m32.558560215s) [seg_pos={0 0}, upload=527821]: Segment reached optimal threshold of 13 pieces. Dec 30 17:41:32.687695706 (602h12m32.558778761s) [noise=true , address=77.90.31.10:4518 , piece_id=WUFRRFHXLNKOWWUPN3GSTOL7SPDJLNCAF6H5N3KT73UTTTHETXPQ, seg_pos={0 0}, upload=527821]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:58848->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:58848->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58848->77.90.31.10:4518: use of closed network connection Dec 30 17:41:32.687704453 (602h12m32.558787488s) [noise=true , address=77.90.31.10:4518 , piece_id=WUFRRFHXLNKOWWUPN3GSTOL7SPDJLNCAF6H5N3KT73UTTTHETXPQ, seg_pos={0 0}, upload=527821]: Long tail cancellation due to optimal threshold Dec 30 17:41:32.687849549 (602h12m32.558932584s) [noise=true , address=145.79.227.35:4481 , piece_id=UPYL65RVDH5Z7RJOOVB7474ZHLK3665T2DPUXIHC2H67CK3Q56EA, seg_pos={0 0}, upload=527821]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 30 17:41:32.687859448 (602h12m32.558942483s) [noise=true , address=145.79.227.35:4481 , piece_id=UPYL65RVDH5Z7RJOOVB7474ZHLK3665T2DPUXIHC2H67CK3Q56EA, seg_pos={0 0}, upload=527821]: Long tail cancellation due to optimal threshold Dec 30 17:41:32.687926505 (602h12m32.559009571s) [noise=true , address=77.90.31.16:4736 , piece_id=T34VZ7QZXCZVXDTLGIVJT3P7ROWJUZKVLA2SWIVD5ZVA7NI5KC5A, seg_pos={0 0}, upload=527821]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:45130->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:45130->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45130->77.90.31.16:4736: use of closed network connection Dec 30 17:41:32.687938358 (602h12m32.559021393s) [noise=true , address=77.90.31.16:4736 , piece_id=T34VZ7QZXCZVXDTLGIVJT3P7ROWJUZKVLA2SWIVD5ZVA7NI5KC5A, seg_pos={0 0}, upload=527821]: Long tail cancellation due to optimal threshold Dec 30 17:41:32.688005506 (602h12m32.559088541s) [seg_pos={0 0}, upload=527821]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 17:41:32.688092161 (602h12m32.559175206s) [upload=527821]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 17:41:32.688097541 (602h12m32.559180586s) [upload=527821]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 17:41:32.984954887 (602h12m32.856037922s) [upload=527821]: Done uploading segments... Dec 30 17:41:32.984976107 (602h12m32.856059142s) [upload=527821]: Upload finished. err: <nil> Duration: 6.353765459s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..X...X.....................................................] duration: 5.966446468s trace: https://jaeger.dev.storj.io/trace/6ea66b1ed6f4c6e5 FAIL (2025-12-30 17:44:27 UTC) orbiter warning: context deadline exceeded Duration: 5.96646289s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 38.146867694s trace: https://jaeger.dev.storj.io/trace/5ea8d7d4f61e13a2 FAIL (2025-12-30 17:44:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1X5A7bLqMEQuaMbf9xZ7PDUD4161sz2QqdM9r59y3bsyPBJhaK, Piece ID: HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ) context deadline exceeded; piecestore: (Node ID: 12eLKUg2vTnZMrLaraSPEoGXe3jDVBJYyURdcn4f2gCN5pvvCQ8, Piece ID: KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ) context deadline exceeded; piecestore: (Node ID: 12W4n6uf6azQk1QgeR8Xw7UkpnZm19zdyFKwojCsqZt2GFHrpce, Piece ID: DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ) context deadline exceeded ======LOGS====== 17.114181822 (601h42m16.985264867s) [noise=true , address=145.79.227.44:4208 , piece_id=3OXDCHZSOIFS5VQCI4SXSAQFICDCYHRE5NGFBERR33KOTGSZWLDQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.114641387 (601h42m16.985724432s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.115652532 (601h42m16.986735578s) [noise=true , address=77.90.31.17:4790 , piece_id=5RSRESKGDA2YF4NBRF4ZFABBKWJK4NO6VLEXW2Z6EY3OPXH2WPNA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.115937134 (601h42m16.987020179s) [noise=true , address=64.49.13.155:4923 , piece_id=MUJD7XLM2YRLRCHYVSDPBE6VBYLHCOA7HJQP22ZSET4N574XQMJA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.116814986 (601h42m16.987898031s) [noise=true , address=45.91.116.11:4139 , piece_id=72DPEWZ7ZPZZ53UGYV2VCUKLCK7HCM22CDBRHWVLJGDGO5B5JADQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.117501182 (601h42m16.988584228s) [noise=true , address=77.90.31.14:4665 , piece_id=DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.118263334 (601h42m16.989346379s) [noise=true , address=145.79.227.40:4084 , piece_id=ZADZDRKBIILLAI4IT335CYETOSFOFUUPQIH6HZJCTHKRTZSPDJUQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.118555850 (601h42m16.989638886s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.118765930 (601h42m16.989849036s) [noise=true , address=77.90.31.21:4905 , piece_id=HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.119029533 (601h42m16.990112568s) [noise=true , address=77.90.31.10:4533 , piece_id=QFBBYNQHQL2G7PEREK43FZBRUMKMVNBC2XKTE2NWTVTKN2452LPA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.119417672 (601h42m16.990500707s) [noise=true , address=45.91.116.12:4258 , piece_id=KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.119441588 (601h42m16.990524643s) [noise=true , address=145.79.227.40:4053 , piece_id=JCSIUMWKEAQNQXTNSJTN6VSB3V2SIDRQK67QMK46B4GOUWEMQE2A, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.119763470 (601h42m16.990846505s) [noise=true , address=77.90.31.18:4821 , piece_id=MXZFKXM3JV2HQURDLIU7BKWLAKDZ27O4ER3SGC7QBZV5I3J2EHKA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.120028685 (601h42m16.99111172s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.120086986 (601h42m16.991170031s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.120586507 (601h42m16.991669562s) [noise=true , address=45.91.116.13:4168 , piece_id=AKUE5H3A2UECQOZY3GV3MWZKIHSNKCDOG32DKCNHIRTJROJ74M3A, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:17.121258768 (601h42m16.992341813s) [noise=true , address=145.79.227.9:4379 , piece_id=QTFXFSGGKYBQSPWJYBBJMQUL3D5EEO3IX4C53ECD7FXMUOGUBC7Q, seg_pos={0 0}, upload=527370]: Uploading piece... Dec 30 17:11:18.568739165 (601h42m18.43982221s) [upload=527370]: Next returned nil segment Dec 30 17:11:18.568751328 (601h42m18.439834363s) [upload=527370]: Waiting for error group managing segments... Dec 30 17:11:18.978424178 (601h42m18.849507213s) [noise=true , address=64.49.13.155:4923 , piece_id=MUJD7XLM2YRLRCHYVSDPBE6VBYLHCOA7HJQP22ZSET4N574XQMJA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:18.986258125 (601h42m18.85734116s) [noise=true , address=145.79.227.9:4379 , piece_id=QTFXFSGGKYBQSPWJYBBJMQUL3D5EEO3IX4C53ECD7FXMUOGUBC7Q, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.274519421 (601h42m19.145602456s) [noise=true , address=77.90.31.14:4665 , piece_id=DR22BOMJOZISUMA7AJ26TPM5UCSQWRBLNFSELMJXS6CCQ7CCXSGQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.315775438 (601h42m19.186858473s) [noise=true , address=45.91.116.13:4168 , piece_id=AKUE5H3A2UECQOZY3GV3MWZKIHSNKCDOG32DKCNHIRTJROJ74M3A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.335765371 (601h42m19.206848406s) [noise=true , address=77.90.31.21:4905 , piece_id=HRBQB7QXPOEU5ZGEB6NPYVZLYGBLJEMRPDKNNCO4QBD473L7LALQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.354812129 (601h42m19.225895164s) [noise=true , address=77.90.31.10:4533 , piece_id=QFBBYNQHQL2G7PEREK43FZBRUMKMVNBC2XKTE2NWTVTKN2452LPA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.392024917 (601h42m19.263107952s) [noise=true , address=45.91.116.11:4139 , piece_id=72DPEWZ7ZPZZ53UGYV2VCUKLCK7HCM22CDBRHWVLJGDGO5B5JADQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.396084327 (601h42m19.267167372s) [noise=true , address=77.90.31.18:4821 , piece_id=MXZFKXM3JV2HQURDLIU7BKWLAKDZ27O4ER3SGC7QBZV5I3J2EHKA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.862099099 (601h42m19.733182134s) [noise=true , address=77.90.31.17:4790 , piece_id=5RSRESKGDA2YF4NBRF4ZFABBKWJK4NO6VLEXW2Z6EY3OPXH2WPNA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.905412913 (601h42m19.776495948s) [noise=true , address=145.79.227.44:4208 , piece_id=3OXDCHZSOIFS5VQCI4SXSAQFICDCYHRE5NGFBERR33KOTGSZWLDQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.910309688 (601h42m19.781392743s) [noise=true , address=145.79.227.30:4794 , piece_id=EJH6Q3T2BMPVLG6F7CW7N4GYOA3EZFPADL6VPT7WKKG65B2WZVXQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:19.934233361 (601h42m19.805316397s) [noise=true , address=145.79.227.40:4084 , piece_id=ZADZDRKBIILLAI4IT335CYETOSFOFUUPQIH6HZJCTHKRTZSPDJUQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:20.074011527 (601h42m19.945094572s) [noise=true , address=45.91.116.12:4258 , piece_id=KSKYRNREZZIOXR6HCQT3KSJVOUUSDFQKU5TSNTQWPH3VXLXJDVQQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:20.106010337 (601h42m19.977093372s) [noise=true , address=145.79.227.40:4053 , piece_id=JCSIUMWKEAQNQXTNSJTN6VSB3V2SIDRQK67QMK46B4GOUWEMQE2A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: <nil> Dec 30 17:11:20.106080911 (601h42m19.977163946s) [seg_pos={0 0}, upload=527370]: Segment reached optimal threshold of 14 pieces. Dec 30 17:11:20.106274520 (601h42m19.977357565s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Tve8WUWhJCK2cVWLbxxnz3WVzR2bi2wLx5XrF6QbhSN4HcsQG): protocol: expected piece hash; context canceled; EOF Dec 30 17:11:20.106281644 (601h42m19.977364679s) [noise=true , address=77.90.31.11:4585 , piece_id=T67YGSBNUYNPW2OLP3EBGDYW4KSHXOISF5HK2JAQ57SHHMWJ4JYA, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold Dec 30 17:11:20.106351607 (601h42m19.977434642s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF Dec 30 17:11:20.106367767 (601h42m19.977450802s) [noise=true , address=145.79.227.49:4367 , piece_id=56VRUHR2QFDW77FUW5HPCF3QMMGGLLV74SE76G32TC62DKRDNO7A, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold Dec 30 17:11:20.106424967 (601h42m19.977508002s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gyQJLnzWmRESyd8m6G7aZWsQwVfqRQNARxRmd7GwwfYwhuzVn): protocol: expected piece hash; context canceled; EOF Dec 30 17:11:20.106432451 (601h42m19.977515486s) [noise=true , address=77.90.31.13:4630 , piece_id=CBZGS5WM4HMTV5UPRIYDWVXN42QXEXE2ITE4H4YIVRDC7LBMWDUA, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold Dec 30 17:11:20.106540666 (601h42m19.977623702s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ijEJgYouyMkUqpLWWz7eH2NGvvs9XpN2ujDjupPykDe5eWcB1): protocol: expected piece hash; context canceled; EOF Dec 30 17:11:20.106546207 (601h42m19.977629242s) [noise=true , address=145.79.227.33:4875 , piece_id=HKX3Z5X4HOHUWWN4RXXXQ4VORLWNDV3TUUC6D7XNZMYD42ZZIROQ, seg_pos={0 0}, upload=527370]: Long tail cancellation due to optimal threshold Dec 30 17:11:20.106563380 (601h42m19.977646415s) [seg_pos={0 0}, upload=527370]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 30 17:11:20.106625529 (601h42m19.977708564s) [upload=527370]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 17:11:20.106629837 (601h42m19.977712882s) [upload=527370]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 17:11:20.406465452 (601h42m20.277548507s) [upload=527370]: Done uploading segments... Dec 30 17:11:20.406504206 (601h42m20.277587251s) [upload=527370]: Upload finished. err: <nil> Duration: 38.146886421s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [X.X.X.X....X.XXXXX.XX.X.XXX......XX.XXX...XX..X.....X.XXX...] duration: 5.721771897s trace: https://jaeger.dev.storj.io/trace/42312365b09db431 FAIL (2025-12-30 17:44:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12u52PWsW5g7XTky7sxS5mPFqpMD4AHCXFfera1WZyZF9xf6ma9, Piece ID: 3U772XXF3AB2CZCRKYHSAXLF3B4VW7W2ISZHBUAS3BBSUQF6IDCQ) context deadline exceeded; piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: KF5PY3KZ42T6CFJXFDOOC344DIL6SZANTQ54RAS4RZA6PQMYBGEQ) context deadline exceeded ======LOGS====== g segments... Dec 30 17:41:29.742890691 (602h12m29.613973726s) [upload=527821]: Got next segment. Inline: false Dec 30 17:41:29.742941117 (602h12m29.614024153s) [upload=527821]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 30 17:41:29.742949253 (602h12m29.614032288s) [upload=527821]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 30 17:41:29.994240285 (602h12m29.86532332s) [seg_pos={0 0}, upload=527821]: Begin upload segment... Dec 30 17:41:29.994289349 (602h12m29.865372384s) [seg_pos={0 0}, upload=527821]: Done begin upload segment. Dec 30 17:41:29.994316519 (602h12m29.865399555s) [noise=true , address=145.79.227.45:4231 , piece_id=AJR4RUB6DWJRXPN3U5GVN556R4QFWDEJCTUAP6A4FYHPCV67OCSQ, seg_pos={0 0}, upload=527821]: Uploading piece... Dec 30 17:41:29.994414386 (602h12m29.865497421s) [noise=true , address=145.79.227.46:4603 , piece_id=DBKR4JT3SVCLF5RDVFC4XNIYOY7SW5KPLLZSHJ5E5CMHZMJ4VQCQ, seg_pos={0 0}, upload=527821]: Uploading piece... Dec 30 17:41:29.994888609 (602h12m29.865971644s) [noise=true , address=45.91.116.6:4975 , piece_id=KF5PY3KZ42T6CFJXFDOOC344DIL6SZANTQ54RAS4RZA6PQMYBGEQ, seg_pos={0 0}, upload=527821]: Uploading piece... Dec 30 17:41:29.995193370 (602h12m29.866276405s) [noise=true , address=145.79.227.39:4014 , piece_id=BNWQMRUHNUN5P3ICNNOQWFIY4UYLT2A73BVBPPJS42AUJDHJAHGQ, seg_pos={0 0}, upload=527821]: Uploading piece... Dec 30 17:41:29.995701447 (602h12m29.866784482s) [noise=true , address=145.79.227.52:4446 , piece_id=DT3MQLERNJLSKDDUI5XXI33RWHDFMRQ4Q74VFY5CBKLABSBAQXYA, seg_pos={0 0}, upload=527821]: Uploading piece... Dec 30 17:41:29.996093293 (602h12m29.867176328s) [noise=true , address=45.91.116.13:4153 , piece_id=Q2XJ6SJVS7O7T6JRWIDLDQTADF7IM3JFOBRGYTDAYCPDH3IYIEOQ, seg_pos={0 0}, upload=527821]: Uploading piece... Dec 30 17:41:29.999308716 (602h12m29.870391751s) [noise=true , address=145.79.227.32:4834 , piece_id=IJ36JBCLTCF3OYU73E24VW7IOMYYD6YZMK6CRYFFNUDZMNA3Z5KA, seg_pos={0 0}, upload=527821]: Uploading piece... Dec 30 17:41:30.000590808 (602h12m29.871673863s) [noise=true , address=145.79.227.17:4970 , piece_id=FN2HC3KAZNAYHHGE3QDSHT6RDEP6YRRUAPUDLEHFPJKCZ2LT4FOA, seg_pos={0 0}, upload=527821]: Uploading piece... Dec 30 17:41:30.002162009 (602h12m29.873245074s) [noise=true , address=145.79.227.27:4691 , piece_id=CYIOOBJCMRFBL6ZNRXMB7S7P5XDWGUHVPHXATEJ7BVGLFGDN3PKQ, seg_pos={0 0}, upload=527821]: Uploading piece... Dec 30 17:41:30.003956816 (602h12m29.875039871s) [noise=true , address=145.79.227.41:4085 , piece_id=ZY5QJKXK33T6UK3AEYTVX2KQFXXUUVVK5BH73DE6TNKKDJVJGNGQ, seg_pos={0 0}, upload=527821]: Uploading piece... Dec 30 17:41:30.005277872 (602h12m29.876360917s) [noise=true , address=145.79.227.43:4158 , piece_id=GHXYRFMQKKQYF3TTT44BNM7MXD44FEKLLA5Q55SRD2AZ3YXFBJPQ, seg_pos={0 0}, upload=527821]: Uploading piece... Dec 30 17:41:30.006712503 (602h12m29.877795539s) [noise=true , address=77.90.31.10:4518 , piece_id=WUFRRFHXLNKOWWUPN3GSTOL7SPDJLNCAF6H5N3KT73UTTTHETXPQ, seg_pos={0 0}, upload=527821]: Uploading piece... Dec 30 17:41:30.007057100 (602h12m29.878140135s) [noise=true , address=145.79.227.47:4266 , piece_id=ZCM557E5SSQPU3Q3RVAOHU4ZXIAZMPPKARICBO7WW6SNIY3B5TSA, seg_pos={0 0}, upload=527821]: Uploading piece... Dec 30 17:41:30.008176651 (602h12m29.879259696s) [noise=true , address=145.79.227.35:4481 , piece_id=UPYL65RVDH5Z7RJOOVB7474ZHLK3665T2DPUXIHC2H67CK3Q56EA, seg_pos={0 0}, upload=527821]: Uploading piece... Dec 30 17:41:30.008429382 (602h12m29.879512427s) [noise=true , address=77.90.31.14:4664 , piece_id=3U772XXF3AB2CZCRKYHSAXLF3B4VW7W2ISZHBUAS3BBSUQF6IDCQ, seg_pos={0 0}, upload=527821]: Uploading piece... Dec 30 17:41:30.009714610 (602h12m29.880797645s) [noise=true , address=77.90.31.16:4736 , piece_id=T34VZ7QZXCZVXDTLGIVJT3P7ROWJUZKVLA2SWIVD5ZVA7NI5KC5A, seg_pos={0 0}, upload=527821]: Uploading piece... Dec 30 17:41:31.711933153 (602h12m31.583016198s) [upload=527821]: Next returned nil segment Dec 30 17:41:31.711961837 (602h12m31.583044882s) [upload=527821]: Waiting for error group managing segments... Dec 30 17:41:31.999912104 (602h12m31.870995149s) [noise=true , address=145.79.227.39:4014 , piece_id=BNWQMRUHNUN5P3ICNNOQWFIY4UYLT2A73BVBPPJS42AUJDHJAHGQ, seg_pos={0 0}, upload=527821]: Done uploading piece. err: <nil> Dec 30 17:41:32.007187788 (602h12m31.878270833s) [noise=true , address=145.79.227.52:4446 , piece_id=DT3MQLERNJLSKDDUI5XXI33RWHDFMRQ4Q74VFY5CBKLABSBAQXYA, seg_pos={0 0}, upload=527821]: Done uploading piece. err: <nil> Dec 30 17:41:32.009116290 (602h12m31.880199335s) [noise=true , address=145.79.227.45:4231 , piece_id=AJR4RUB6DWJRXPN3U5GVN556R4QFWDEJCTUAP6A4FYHPCV67OCSQ, seg_pos={0 0}, upload=527821]: Done uploading piece. err: <nil> Dec 30 17:41:32.011104976 (602h12m31.882188011s) [noise=true , address=145.79.227.32:4834 , piece_id=IJ36JBCLTCF3OYU73E24VW7IOMYYD6YZMK6CRYFFNUDZMNA3Z5KA, seg_pos={0 0}, upload=527821]: Done uploading piece. err: <nil> Dec 30 17:41:32.016382305 (602h12m31.88746534s) [noise=true , address=145.79.227.47:4266 , piece_id=ZCM557E5SSQPU3Q3RVAOHU4ZXIAZMPPKARICBO7WW6SNIY3B5TSA, seg_pos={0 0}, upload=527821]: Done uploading piece. err: <nil> Dec 30 17:41:32.022409591 (602h12m31.893492636s) [noise=true , address=145.79.227.41:4085 , piece_id=ZY5QJKXK33T6UK3AEYTVX2KQFXXUUVVK5BH73DE6TNKKDJVJGNGQ, seg_pos={0 0}, upload=527821]: Done uploading piece. err: <nil> Dec 30 17:41:32.082258722 (602h12m31.953341757s) [noise=true , address=145.79.227.46:4603 , piece_id=DBKR4JT3SVCLF5RDVFC4XNIYOY7SW5KPLLZSHJ5E5CMHZMJ4VQCQ, seg_pos={0 0}, upload=527821]: Done uploading piece. err: <nil> Dec 30 17:41:32.093969341 (602h12m31.965052386s) [noise=true , address=145.79.227.27:4691 , piece_id=CYIOOBJCMRFBL6ZNRXMB7S7P5XDWGUHVPHXATEJ7BVGLFGDN3PKQ, seg_pos={0 0}, upload=527821]: Done uploading piece. err: <nil> Dec 30 17:41:32.106393409 (602h12m31.977476444s) [noise=true , address=145.79.227.17:4970 , piece_id=FN2HC3KAZNAYHHGE3QDSHT6RDEP6YRRUAPUDLEHFPJKCZ2LT4FOA, seg_pos={0 0}, upload=527821]: Done uploading piece. err: <nil> Dec 30 17:41:32.281425142 (602h12m32.152508177s) [noise=true , address=77.90.31.14:4664 , piece_id=3U772XXF3AB2CZCRKYHSAXLF3B4VW7W2ISZHBUAS3BBSUQF6IDCQ, seg_pos={0 0}, upload=527821]: Done uploading piece. err: <nil> Dec 30 17:41:32.327920055 (602h12m32.1990031s) [noise=true , address=45.91.116.6:4975 , piece_id=KF5PY3KZ42T6CFJXFDOOC344DIL6SZANTQ54RAS4RZA6PQMYBGEQ, seg_pos={0 0}, upload=527821]: Done uploading piece. err: <nil> Dec 30 17:41:32.466119095 (602h12m32.33720214s) [noise=true , address=45.91.116.13:4153 , piece_id=Q2XJ6SJVS7O7T6JRWIDLDQTADF7IM3JFOBRGYTDAYCPDH3IYIEOQ, seg_pos={0 0}, upload=527821]: Done uploading piece. err: <nil> Dec 30 17:41:32.687451991 (602h12m32.558535037s) [noise=true , address=145.79.227.43:4158 , piece_id=GHXYRFMQKKQYF3TTT44BNM7MXD44FEKLLA5Q55SRD2AZ3YXFBJPQ, seg_pos={0 0}, upload=527821]: Done uploading piece. err: <nil> Dec 30 17:41:32.687477170 (602h12m32.558560215s) [seg_pos={0 0}, upload=527821]: Segment reached optimal threshold of 13 pieces. Dec 30 17:41:32.687695706 (602h12m32.558778761s) [noise=true , address=77.90.31.10:4518 , piece_id=WUFRRFHXLNKOWWUPN3GSTOL7SPDJLNCAF6H5N3KT73UTTTHETXPQ, seg_pos={0 0}, upload=527821]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: write tcp 188.166.225.170:58848->77.90.31.10:4518: use of closed network connection; write tcp 188.166.225.170:58848->77.90.31.10:4518: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58848->77.90.31.10:4518: use of closed network connection Dec 30 17:41:32.687704453 (602h12m32.558787488s) [noise=true , address=77.90.31.10:4518 , piece_id=WUFRRFHXLNKOWWUPN3GSTOL7SPDJLNCAF6H5N3KT73UTTTHETXPQ, seg_pos={0 0}, upload=527821]: Long tail cancellation due to optimal threshold Dec 30 17:41:32.687849549 (602h12m32.558932584s) [noise=true , address=145.79.227.35:4481 , piece_id=UPYL65RVDH5Z7RJOOVB7474ZHLK3665T2DPUXIHC2H67CK3Q56EA, seg_pos={0 0}, upload=527821]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 30 17:41:32.687859448 (602h12m32.558942483s) [noise=true , address=145.79.227.35:4481 , piece_id=UPYL65RVDH5Z7RJOOVB7474ZHLK3665T2DPUXIHC2H67CK3Q56EA, seg_pos={0 0}, upload=527821]: Long tail cancellation due to optimal threshold Dec 30 17:41:32.687926505 (602h12m32.559009571s) [noise=true , address=77.90.31.16:4736 , piece_id=T34VZ7QZXCZVXDTLGIVJT3P7ROWJUZKVLA2SWIVD5ZVA7NI5KC5A, seg_pos={0 0}, upload=527821]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: write tcp 188.166.225.170:45130->77.90.31.16:4736: use of closed network connection; write tcp 188.166.225.170:45130->77.90.31.16:4736: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45130->77.90.31.16:4736: use of closed network connection Dec 30 17:41:32.687938358 (602h12m32.559021393s) [noise=true , address=77.90.31.16:4736 , piece_id=T34VZ7QZXCZVXDTLGIVJT3P7ROWJUZKVLA2SWIVD5ZVA7NI5KC5A, seg_pos={0 0}, upload=527821]: Long tail cancellation due to optimal threshold Dec 30 17:41:32.688005506 (602h12m32.559088541s) [seg_pos={0 0}, upload=527821]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 30 17:41:32.688092161 (602h12m32.559175206s) [upload=527821]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 30 17:41:32.688097541 (602h12m32.559180586s) [upload=527821]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 30 17:41:32.984954887 (602h12m32.856037922s) [upload=527821]: Done uploading segments... Dec 30 17:41:32.984976107 (602h12m32.856059142s) [upload=527821]: Upload finished. err: <nil> Duration: 5.721802144s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 12 out of 60 is failed (threshold=6) [......X..X....X.XX..X..X......X.........X..XXX..............] duration: 3.621542586s trace: https://jaeger.dev.storj.io/trace/56692ebd5ab65c7c FAIL (2025-12-30 17:43:57 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.621653389s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [...X...X....................................................] duration: 5.744699764s trace: https://jaeger.dev.storj.io/trace/6ea66b1ed6f4c6e5 FAIL (2025-12-30 17:43:41 UTC) orbiter warning: context deadline exceeded Duration: 5.744716446s core uplink us-select-1 us1-placement-20

1
...
223
224
225
226
227
...
7152