Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXX.X.X.X................................XXXXXXXX] duration: 30.00082316s FAIL (2025-08-07 22:18:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== (12h30m55.075777221s) [noise=true , address=77.90.31.11:4555 , piece_id=A6WWFITSRKE4GVTQODOYU36ORK7OVE3MVVR7B4FPSSP64DATOQ6A, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.522908903 (12h30m55.11225389s) [noise=true , address=77.90.31.18:4805 , piece_id=VQ3CGKCPEKYPCAIOHQ2FZSINJBHBCCQUZD2AD4AIHX72SZN7SEKA, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.546118308 (12h30m55.135463295s) [noise=true , address=104.36.82.4:4802 , piece_id=B4Q65J747GHE2ZRZDM7RLWC3AQC7TLK5D53REAUA5DMHT3TGA44Q, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.568801126 (12h30m55.158146113s) [noise=true , address=104.36.82.4:4804 , piece_id=BI6ZPR56BKHTA2QGM3BNUBFJM6PAD4K27P5TYHRGLG4NBD56LC4A, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.587208293 (12h30m55.176553341s) [noise=true , address=77.90.31.17:4770 , piece_id=BG2ZL7XC52NEJC5FRS4V75MEPJH34B27IPLTJNQLSX4N3OZGVHXA, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.587245686 (12h30m55.176590713s) [seg_pos={0 0}, upload=13000]: Segment reached optimal threshold of 13 pieces. Aug 7 22:10:55.587371516 (12h30m55.176716533s) [noise=true , address=45.91.116.4:4939 , piece_id=FQ4Z6Y4KIOGJLPCZFVPCUJSYOMGDOTZNAIEKGNWKASCLKOWLCTUQ, seg_pos={0 0}, upload=13000]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Aug 7 22:10:55.587442001 (12h30m55.176786988s) [noise=true , address=45.91.116.7:4009 , piece_id=47X2WBQ4XYRNAZDBVU572QSRHXUNSYRCOUALM4SF4WRHCDW6QTCA, seg_pos={0 0}, upload=13000]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Aug 7 22:10:55.587644678 (12h30m55.176989726s) [noise=true , address=104.36.82.12:4150 , piece_id=XG7E3FHDR2VCPREGTWFFJ52JLGUPRUKQJ75KFYEUXPI4OUNTB3ZA, seg_pos={0 0}, upload=13000]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: context canceled; context canceled Aug 7 22:10:55.587681550 (12h30m55.177026557s) [seg_pos={0 0}, upload=13000]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 22:10:55.587777072 (12h30m55.177122109s) [upload=13000]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:10:55.587787912 (12h30m55.177132929s) [upload=13000]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:55.887211865 (12h30m55.476556883s) [upload=13000]: Done uploading segments... Aug 7 22:10:55.887228857 (12h30m55.476573844s) [upload=13000]: Upload finished. err: <nil> Aug 7 22:11:40.872949328 (12h31m40.462294315s) [upload=13014]: Starting upload Aug 7 22:11:40.873003161 (12h31m40.462348148s) [upload=13014]: Done starting upload Aug 7 22:11:40.873072924 (12h31m40.462417911s) [upload=13014]: Uploading segments... Aug 7 22:11:40.873431210 (12h31m40.462776287s) [upload=13014]: Got next segment. Inline: false Aug 7 22:11:40.873471537 (12h31m40.462816525s) [upload=13014]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:11:40.873480314 (12h31m40.462825331s) [upload=13014]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:12:10.229450117 (12h32m9.818795124s) [upload=13014]: Next segment err: context deadline exceeded Aug 7 22:12:10.229589654 (12h32m9.818934631s) [upload=13014]: Done uploading segments... Aug 7 22:12:10.229603972 (12h32m9.818948949s) [upload=13014]: Upload finished. err: context deadline exceeded Aug 7 22:12:48.993762773 (12h32m48.58310776s) [upload=13033]: Starting upload Aug 7 22:12:48.993844269 (12h32m48.583189246s) [upload=13033]: Done starting upload Aug 7 22:12:48.993904404 (12h32m48.583249391s) [upload=13033]: Uploading segments... Aug 7 22:12:48.994022439 (12h32m48.583367427s) [upload=13033]: Got next segment. Inline: false Aug 7 22:12:48.994048309 (12h32m48.583393296s) [upload=13033]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:48.994052647 (12h32m48.583397634s) [upload=13033]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:13:18.347381136 (12h33m17.936726133s) [upload=13033]: Next segment err: context deadline exceeded Aug 7 22:13:18.347514681 (12h33m17.936859668s) [upload=13033]: Done uploading segments... Aug 7 22:13:18.347529670 (12h33m17.936874657s) [upload=13033]: Upload finished. err: context deadline exceeded Aug 7 22:13:51.696413792 (12h33m51.285758769s) [upload=13052]: Starting upload Aug 7 22:13:51.696517580 (12h33m51.285862567s) [upload=13052]: Done starting upload Aug 7 22:13:51.696601842 (12h33m51.285946829s) [upload=13052]: Uploading segments... Aug 7 22:13:51.696787407 (12h33m51.286132414s) [upload=13052]: Got next segment. Inline: false Aug 7 22:13:51.696823415 (12h33m51.286168413s) [upload=13052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:13:51.696829066 (12h33m51.286174073s) [upload=13052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:14:21.042712175 (12h34m20.632057152s) [upload=13052]: Next segment err: context deadline exceeded Aug 7 22:14:21.042793340 (12h34m20.632138327s) [upload=13052]: Done uploading segments... Aug 7 22:14:21.042809000 (12h34m20.632153987s) [upload=13052]: Upload finished. err: context deadline exceeded Aug 7 22:14:36.252577987 (12h34m35.841922974s) [upload=13065]: Starting upload Aug 7 22:14:36.252631760 (12h34m35.841976747s) [upload=13065]: Done starting upload Aug 7 22:14:36.252692656 (12h34m35.842037643s) [upload=13065]: Uploading segments... Aug 7 22:14:36.252805532 (12h34m35.842150519s) [upload=13065]: Got next segment. Inline: false Aug 7 22:14:36.252856870 (12h34m35.842201857s) [upload=13065]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:36.252863202 (12h34m35.842208189s) [upload=13065]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:15:05.607209972 (12h35m5.196554959s) [upload=13065]: Next segment err: context deadline exceeded Aug 7 22:15:05.607234078 (12h35m5.196579065s) [upload=13065]: Done uploading segments... Aug 7 22:15:05.607262002 (12h35m5.196606989s) [upload=13065]: Upload finished. err: context deadline exceeded Aug 7 22:15:30.794674976 (12h35m30.384019963s) [upload=13077]: Starting upload Aug 7 22:15:30.794787000 (12h35m30.384132008s) [upload=13077]: Done starting upload Aug 7 22:15:30.794881030 (12h35m30.384226007s) [upload=13077]: Uploading segments... Aug 7 22:15:30.795138724 (12h35m30.384483711s) [upload=13077]: Got next segment. Inline: false Aug 7 22:15:30.795218847 (12h35m30.384563834s) [upload=13077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:30.795225099 (12h35m30.384570106s) [upload=13077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:16:00.147601500 (12h35m59.736946487s) [upload=13077]: Next segment err: context deadline exceeded Aug 7 22:16:00.147674640 (12h35m59.737019628s) [upload=13077]: Done uploading segments... Aug 7 22:16:00.147688897 (12h35m59.737033884s) [upload=13077]: Upload finished. err: context deadline exceeded Aug 7 22:16:34.181460693 (12h36m33.77080568s) [upload=13093]: Starting upload Aug 7 22:16:34.181519996 (12h36m33.770864983s) [upload=13093]: Done starting upload Aug 7 22:16:34.181628824 (12h36m33.770973811s) [upload=13093]: Uploading segments... Aug 7 22:16:34.181727263 (12h36m33.77107226s) [upload=13093]: Got next segment. Inline: false Aug 7 22:16:34.181766387 (12h36m33.771111385s) [upload=13093]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:16:34.181772269 (12h36m33.771117267s) [upload=13093]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:17:03.528495677 (12h37m3.117840664s) [upload=13093]: Next segment err: context deadline exceeded Aug 7 22:17:03.528680499 (12h37m3.118025497s) [upload=13093]: Done uploading segments... Aug 7 22:17:03.528713574 (12h37m3.118058551s) [upload=13093]: Upload finished. err: context deadline exceeded Aug 7 22:17:08.817549216 (12h37m8.406894193s) [upload=13098]: Starting upload Aug 7 22:17:08.817602037 (12h37m8.406947024s) [upload=13098]: Done starting upload Aug 7 22:17:08.817716647 (12h37m8.407061634s) [upload=13098]: Uploading segments... Aug 7 22:17:08.817823451 (12h37m8.407168428s) [upload=13098]: Got next segment. Inline: false Aug 7 22:17:08.817862286 (12h37m8.407207273s) [upload=13098]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:17:08.817866333 (12h37m8.407211311s) [upload=13098]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:17:38.170422707 (12h37m37.759767684s) [upload=13098]: Next segment err: context deadline exceeded Aug 7 22:17:38.170520234 (12h37m37.759865211s) [upload=13098]: Done uploading segments... Aug 7 22:17:38.170546113 (12h37m37.7598911s) [upload=13098]: Upload finished. err: context deadline exceeded Aug 7 22:17:39.060431005 (12h37m38.649775992s) [upload=13107]: Starting upload Aug 7 22:17:39.060493555 (12h37m38.649838542s) [upload=13107]: Done starting upload Aug 7 22:17:39.060548590 (12h37m38.649893577s) [upload=13107]: Uploading segments... Aug 7 22:17:39.060739074 (12h37m38.650084061s) [upload=13107]: Got next segment. Inline: false Aug 7 22:17:39.060826201 (12h37m38.650171188s) [upload=13107]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:17:39.060833395 (12h37m38.650178383s) [upload=13107]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:18:08.415770985 (12h38m8.005115962s) [upload=13107]: Next segment err: context deadline exceeded Aug 7 22:18:08.415875165 (12h38m8.005220202s) [upload=13107]: Done uploading segments... Aug 7 22:18:08.415894011 (12h38m8.005238998s) [upload=13107]: Upload finished. err: context deadline exceeded Duration: 30.000832498s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 23 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXX.....X................................XXXX] duration: 3m4.772366815s FAIL (2025-08-07 22:18:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== b.BatchRequestItem_ObjectBegin Aug 7 22:15:04.181202282 (12h35m3.770547269s) [upload=13072]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:15:40.434664664 (12h35m40.024009671s) [seg_pos={0 0}, upload=13072]: Begin upload segment... Aug 7 22:15:40.434770486 (12h35m40.024115513s) [seg_pos={0 0}, upload=13072]: Done begin upload segment. Aug 7 22:15:40.434816384 (12h35m40.024161372s) [noise=true , address=104.36.82.61:4408 , piece_id=ODDEIQKAHTZE7CQAZ5VYP6J2L42QPTJB4HWRG7X7A67QNHA6KW6A, seg_pos={0 0}, upload=13072]: Uploading piece... Aug 7 22:15:40.434838537 (12h35m40.024183604s) [noise=true , address=104.36.82.59:4312 , piece_id=OVTA43HV7CMKDEAFZMTYWPFIQWHHHNLSFOW6DY5B6MK4XNDRN55A, seg_pos={0 0}, upload=13072]: Uploading piece... Aug 7 22:15:40.434934831 (12h35m40.024279818s) [noise=true , address=104.36.82.5:4855 , piece_id=SO6F72QK6ZOMKQQY2JDU4KEXQMLLTXTTQGA7T7RIAOZ5XJQPQADA, seg_pos={0 0}, upload=13072]: Uploading piece... Aug 7 22:15:40.435355525 (12h35m40.024700512s) [noise=true , address=104.36.82.11:4138 , piece_id=423YZO7WADRWZW54K4DLMCZQZLKGJBWAXQVNGUBJGIFCLKBDXC4Q, seg_pos={0 0}, upload=13072]: Uploading piece... Aug 7 22:15:40.435498369 (12h35m40.024843346s) [noise=true , address=104.36.82.47:4530 , piece_id=OA3KJHXANETB2X3P32YWJ7L3LNBKYSJI4U36BIWN5ZV62FYEFBJA, seg_pos={0 0}, upload=13072]: Uploading piece... Aug 7 22:15:40.435615473 (12h35m40.02496046s) [noise=true , address=104.36.82.16:4318 , piece_id=5AIK4YHDTASBPRRJK52ACBSTF3YDYMOEERRYX5UZ6XAMEQEZVH4A, seg_pos={0 0}, upload=13072]: Uploading piece... Aug 7 22:15:40.435787562 (12h35m40.025132539s) [noise=true , address=104.36.82.5:4845 , piece_id=5PI3CLRG7RFUZ45GRIDBGC7TIQ5KCWQP7QFWXQX4WIALW3SE6LIA, seg_pos={0 0}, upload=13072]: Uploading piece... Aug 7 22:15:40.435939963 (12h35m40.02528495s) [noise=true , address=45.91.116.7:4017 , piece_id=3UJ6J6LO3QUOZLZ7Y23G7UUA6GTITZPBUNU6MTBOCSR33A7MF4MQ, seg_pos={0 0}, upload=13072]: Uploading piece... Aug 7 22:15:40.436103947 (12h35m40.025448934s) [noise=true , address=104.36.82.14:4246 , piece_id=B6YQCB5W7JBVTOAMWRS2RHWQ6YRHPYPHFXZKKZZFRM5I22ZHXYYA, seg_pos={0 0}, upload=13072]: Uploading piece... Aug 7 22:15:40.436168812 (12h35m40.025513799s) [noise=true , address=64.49.13.27:4965 , piece_id=JYYCQIRTG6YME6PRD6K6KBJS4FBIR4IW5MIHHFEWM3RNRTAGBHWQ, seg_pos={0 0}, upload=13072]: Uploading piece... Aug 7 22:15:40.436317235 (12h35m40.025662213s) [noise=true , address=104.36.82.23:4446 , piece_id=PN6WA2SNM2YRYE2HJT55EJCS6EJ5WNCQPZPQM6PX4PKGFZC44LPQ, seg_pos={0 0}, upload=13072]: Uploading piece... Aug 7 22:15:40.436403410 (12h35m40.025748407s) [noise=true , address=64.49.13.137:4499 , piece_id=4OGMMSTT7PN2QNLGZZ57IYUD22VSQS7CHWYO2AQIRNHS4LDIKT6A, seg_pos={0 0}, upload=13072]: Uploading piece... Aug 7 22:15:40.436523881 (12h35m40.025868868s) [noise=true , address=104.36.82.6:4884 , piece_id=KBL444F5N2IKJNLN425LN4UPZCKQIE6K57ZV3HT34L4JBGGNZHMA, seg_pos={0 0}, upload=13072]: Uploading piece... Aug 7 22:15:40.436794669 (12h35m40.026139656s) [noise=true , address=77.90.31.18:4818 , piece_id=3XZ436OJVZKX3NGFGTGNMS2G7D7EVIGABFHZ3TPAMAE4JJC3ZGZQ, seg_pos={0 0}, upload=13072]: Uploading piece... Aug 7 22:15:40.436802424 (12h35m40.026147411s) [noise=true , address=104.36.82.62:4421 , piece_id=O6PK77HVX3HDNJQNHNND22EF5Y5SJ5Q4GJKWSYTI4GP47NGGJGOQ, seg_pos={0 0}, upload=13072]: Uploading piece... Aug 7 22:15:40.436998459 (12h35m40.026343446s) [noise=true , address=104.36.82.46:4490 , piece_id=NSXCTW4PUAAEGXK7PCZDXCXUZEBYAKPFMT52GPYNHIND47JS5ZFA, seg_pos={0 0}, upload=13072]: Uploading piece... Aug 7 22:15:40.438552863 (12h35m40.02789784s) [noise=true , address=64.49.13.140:4515 , piece_id=JIGCKLUEAO6H3JJAW4MPTO3PCESBBFWEZMLPUSZALPKS3II62L3A, seg_pos={0 0}, upload=13072]: Uploading piece... Aug 7 22:15:40.438642985 (12h35m40.027987972s) [noise=true , address=104.36.82.37:4660 , piece_id=QOOMLP4CKM6FNBP25CDMHQ7PGN6ETZDGX4DVFKGEDAETJAGZTH4A, seg_pos={0 0}, upload=13072]: Uploading piece... Aug 7 22:15:41.560077520 (12h35m41.149422508s) [upload=13072]: Next returned nil segment Aug 7 22:15:41.560091408 (12h35m41.149436395s) [upload=13072]: Waiting for error group managing segments... Aug 7 22:15:41.861120730 (12h35m41.450465717s) [noise=true , address=64.49.13.140:4515 , piece_id=JIGCKLUEAO6H3JJAW4MPTO3PCESBBFWEZMLPUSZALPKS3II62L3A, seg_pos={0 0}, upload=13072]: Done uploading piece. err: <nil> Aug 7 22:15:41.945862152 (12h35m41.535207139s) [noise=true , address=64.49.13.137:4499 , piece_id=4OGMMSTT7PN2QNLGZZ57IYUD22VSQS7CHWYO2AQIRNHS4LDIKT6A, seg_pos={0 0}, upload=13072]: Done uploading piece. err: <nil> Aug 7 22:15:41.946698792 (12h35m41.536043779s) [noise=true , address=64.49.13.27:4965 , piece_id=JYYCQIRTG6YME6PRD6K6KBJS4FBIR4IW5MIHHFEWM3RNRTAGBHWQ, seg_pos={0 0}, upload=13072]: Done uploading piece. err: <nil> Aug 7 22:15:42.403195846 (12h35m41.992540823s) [noise=true , address=104.36.82.6:4884 , piece_id=KBL444F5N2IKJNLN425LN4UPZCKQIE6K57ZV3HT34L4JBGGNZHMA, seg_pos={0 0}, upload=13072]: Done uploading piece. err: <nil> Aug 7 22:15:42.499147316 (12h35m42.088492303s) [noise=true , address=104.36.82.11:4138 , piece_id=423YZO7WADRWZW54K4DLMCZQZLKGJBWAXQVNGUBJGIFCLKBDXC4Q, seg_pos={0 0}, upload=13072]: Done uploading piece. err: <nil> Aug 7 22:15:42.506574856 (12h35m42.095919843s) [noise=true , address=104.36.82.61:4408 , piece_id=ODDEIQKAHTZE7CQAZ5VYP6J2L42QPTJB4HWRG7X7A67QNHA6KW6A, seg_pos={0 0}, upload=13072]: Done uploading piece. err: <nil> Aug 7 22:15:42.508597426 (12h35m42.097942413s) [noise=true , address=104.36.82.37:4660 , piece_id=QOOMLP4CKM6FNBP25CDMHQ7PGN6ETZDGX4DVFKGEDAETJAGZTH4A, seg_pos={0 0}, upload=13072]: Done uploading piece. err: <nil> Aug 7 22:15:42.524355996 (12h35m42.113700983s) [noise=true , address=104.36.82.14:4246 , piece_id=B6YQCB5W7JBVTOAMWRS2RHWQ6YRHPYPHFXZKKZZFRM5I22ZHXYYA, seg_pos={0 0}, upload=13072]: Done uploading piece. err: <nil> Aug 7 22:15:42.689053153 (12h35m42.27839814s) [noise=true , address=77.90.31.18:4818 , piece_id=3XZ436OJVZKX3NGFGTGNMS2G7D7EVIGABFHZ3TPAMAE4JJC3ZGZQ, seg_pos={0 0}, upload=13072]: Done uploading piece. err: <nil> Aug 7 22:15:42.990593694 (12h35m42.579938681s) [noise=true , address=45.91.116.7:4017 , piece_id=3UJ6J6LO3QUOZLZ7Y23G7UUA6GTITZPBUNU6MTBOCSR33A7MF4MQ, seg_pos={0 0}, upload=13072]: Done uploading piece. err: <nil> Aug 7 22:15:44.618666775 (12h35m44.208011762s) [noise=true , address=104.36.82.5:4855 , piece_id=SO6F72QK6ZOMKQQY2JDU4KEXQMLLTXTTQGA7T7RIAOZ5XJQPQADA, seg_pos={0 0}, upload=13072]: Done uploading piece. err: <nil> Aug 7 22:15:44.670537936 (12h35m44.259882923s) [noise=true , address=104.36.82.16:4318 , piece_id=5AIK4YHDTASBPRRJK52ACBSTF3YDYMOEERRYX5UZ6XAMEQEZVH4A, seg_pos={0 0}, upload=13072]: Done uploading piece. err: <nil> Aug 7 22:15:44.744425462 (12h35m44.333770449s) [noise=true , address=104.36.82.47:4530 , piece_id=OA3KJHXANETB2X3P32YWJ7L3LNBKYSJI4U36BIWN5ZV62FYEFBJA, seg_pos={0 0}, upload=13072]: Done uploading piece. err: <nil> Aug 7 22:15:45.132772818 (12h35m44.722117805s) [noise=true , address=104.36.82.5:4845 , piece_id=5PI3CLRG7RFUZ45GRIDBGC7TIQ5KCWQP7QFWXQX4WIALW3SE6LIA, seg_pos={0 0}, upload=13072]: Done uploading piece. err: <nil> Aug 7 22:15:45.132798427 (12h35m44.722143414s) [seg_pos={0 0}, upload=13072]: Segment reached optimal threshold of 14 pieces. Aug 7 22:15:45.133004622 (12h35m44.722349599s) [noise=true , address=104.36.82.23:4446 , piece_id=PN6WA2SNM2YRYE2HJT55EJCS6EJ5WNCQPZPQM6PX4PKGFZC44LPQ, seg_pos={0 0}, upload=13072]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1RNJPWzogZpsQfuFzJYZiz8bbHSRSrgSE5FLyhCynSwEDfMid7): protocol: expected piece hash; context canceled; EOF Aug 7 22:15:45.133116205 (12h35m44.722461182s) [noise=true , address=104.36.82.59:4312 , piece_id=OVTA43HV7CMKDEAFZMTYWPFIQWHHHNLSFOW6DY5B6MK4XNDRN55A, seg_pos={0 0}, upload=13072]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12DY4TMNpTfCcXxVyy8HJpbMFZT64wEZiPSysost4V4dbtqHWvY): protocol: rpc: dial tcp 104.36.82.59:4312: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:15:45.133170208 (12h35m44.722515195s) [noise=true , address=104.36.82.62:4421 , piece_id=O6PK77HVX3HDNJQNHNND22EF5Y5SJ5Q4GJKWSYTI4GP47NGGJGOQ, seg_pos={0 0}, upload=13072]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12NEpwAXnTuoC8qPDdDEeRki845XdWxYw2YW46XoFPXhn3BQwxD): protocol: write tcp 188.166.225.170:36088->104.36.82.62:4421: use of closed network connection; write tcp 188.166.225.170:36088->104.36.82.62:4421: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:36088->104.36.82.62:4421: use of closed network connection Aug 7 22:15:45.133236956 (12h35m44.722581943s) [noise=true , address=104.36.82.46:4490 , piece_id=NSXCTW4PUAAEGXK7PCZDXCXUZEBYAKPFMT52GPYNHIND47JS5ZFA, seg_pos={0 0}, upload=13072]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12PUxwz3cZkF8e2VkjTXnu1utrFzQRTxdohfd9EvDaLvyejpTji): protocol: read tcp 188.166.225.170:60916->104.36.82.46:4490: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:15:45.133259008 (12h35m44.722603995s) [seg_pos={0 0}, upload=13072]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:15:45.133341636 (12h35m44.722686613s) [upload=13072]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:15:45.133345984 (12h35m44.722690971s) [upload=13072]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:18:08.311358479 (12h38m7.900703466s) [upload=13072]: Done uploading segments... Aug 7 22:18:08.311383737 (12h38m7.900728724s) [upload=13072]: Upload finished. err: metaclient: context deadline exceeded Duration:* 3m4.772382575s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 25 out of 60 is failed (threshold=3) [XXXXXXXXXXXXX...X................................XXXXXXXXXXX] duration: 30.001122355s FAIL (2025-08-07 22:18:08 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.001132694s core uplink us1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 7 out of 60 is failed (threshold=3) [X.XX..X.XX...............................................X..] duration: 229.97072ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 22:18:07 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 229.995186ms edge gateway vhost bgp us1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 19 out of 60 is failed (threshold=3) [XXXXXXXXX.X........................................XXXXXXXXX] duration: 30.000586639s FAIL (2025-08-07 22:17:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.00059653s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 21 out of 60 is failed (threshold=3) [XXXXXXXXXXXX.X...X...................................XXXXXXX] duration: 30.00139651s FAIL (2025-08-07 22:17:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 9:53.131364442 (12h29m52.720709489s) [upload=12982]: Done starting upload Aug 7 22:09:53.131436210 (12h29m52.720781197s) [upload=12982]: Uploading segments... Aug 7 22:09:53.131465205 (12h29m52.720810192s) [upload=12982]: Got next segment. Inline: true Aug 7 22:09:53.131468902 (12h29m52.720813889s) [upload=12982]: Waiting for error group managing segments... Aug 7 22:09:53.131498809 (12h29m52.720843796s) [upload=12982]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:09:53.131503017 (12h29m52.720848004s) [upload=12982]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:09:53.131505141 (12h29m52.720850118s) [upload=12982]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:09:53.419040148 (12h29m53.008385125s) [upload=12982]: Done uploading segments... Aug 7 22:09:53.419058042 (12h29m53.008403029s) [upload=12982]: Upload finished. err: <nil> Aug 7 22:10:12.329552115 (12h30m11.918897102s) [upload=12989]: Starting upload Aug 7 22:10:12.330067942 (12h30m11.919412929s) [upload=12989]: Done starting upload Aug 7 22:10:12.330139077 (12h30m11.919484054s) [upload=12989]: Uploading segments... Aug 7 22:10:12.330186228 (12h30m11.919531255s) [upload=12989]: Got next segment. Inline: true Aug 7 22:10:12.330192710 (12h30m11.919537737s) [upload=12989]: Waiting for error group managing segments... Aug 7 22:10:12.330240762 (12h30m11.919585749s) [upload=12989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:10:12.330248126 (12h30m11.919593113s) [upload=12989]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:10:12.330252083 (12h30m11.91959709s) [upload=12989]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:12.591107606 (12h30m12.180452593s) [upload=12989]: Done uploading segments... Aug 7 22:10:12.591124920 (12h30m12.180469907s) [upload=12989]: Upload finished. err: <nil> Aug 7 22:10:57.486329607 (12h30m57.075674594s) [upload=13003]: Starting upload Aug 7 22:10:57.486414950 (12h30m57.075759927s) [upload=13003]: Done starting upload Aug 7 22:10:57.486491847 (12h30m57.075836834s) [upload=13003]: Uploading segments... Aug 7 22:10:57.486573694 (12h30m57.075918681s) [upload=13003]: Got next segment. Inline: true Aug 7 22:10:57.486578242 (12h30m57.07592322s) [upload=13003]: Waiting for error group managing segments... Aug 7 22:10:57.486645681 (12h30m57.075990658s) [upload=13003]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:10:57.486650361 (12h30m57.075995348s) [upload=13003]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:10:57.486652795 (12h30m57.075997782s) [upload=13003]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:57.751495816 (12h30m57.340840803s) [upload=13003]: Done uploading segments... Aug 7 22:10:57.751515013 (12h30m57.34086s) [upload=13003]: Upload finished. err: <nil> Aug 7 22:11:44.715949867 (12h31m44.305294854s) [upload=13015]: Starting upload Aug 7 22:11:44.716016213 (12h31m44.3053612s) [upload=13015]: Done starting upload Aug 7 22:11:44.716066600 (12h31m44.305411587s) [upload=13015]: Uploading segments... Aug 7 22:11:44.716104322 (12h31m44.305449309s) [upload=13015]: Got next segment. Inline: true Aug 7 22:11:44.716107879 (12h31m44.305452866s) [upload=13015]: Waiting for error group managing segments... Aug 7 22:11:44.716139289 (12h31m44.305484276s) [upload=13015]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:11:44.716142826 (12h31m44.305487803s) [upload=13015]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:11:44.716146493 (12h31m44.30549147s) [upload=13015]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:12:14.067179909 (12h32m13.656524896s) [upload=13015]: Done uploading segments... Aug 7 22:12:14.067200478 (12h32m13.656545465s) [upload=13015]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:12:53.398168829 (12h32m52.987513816s) [upload=13035]: Starting upload Aug 7 22:12:53.398221350 (12h32m52.987566337s) [upload=13035]: Done starting upload Aug 7 22:12:53.398263170 (12h32m52.987608147s) [upload=13035]: Uploading segments... Aug 7 22:12:53.398291284 (12h32m52.987636261s) [upload=13035]: Got next segment. Inline: true Aug 7 22:12:53.398295111 (12h32m52.987640088s) [upload=13035]: Waiting for error group managing segments... Aug 7 22:12:53.398338094 (12h32m52.987683081s) [upload=13035]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:53.398344366 (12h32m52.987689363s) [upload=13035]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:12:53.398348754 (12h32m52.987693771s) [upload=13035]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:13:22.757670742 (12h33m22.347015729s) [upload=13035]: Done uploading segments... Aug 7 22:13:22.757697282 (12h33m22.34704227s) [upload=13035]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:14:04.336472471 (12h34m3.925817458s) [upload=13057]: Starting upload Aug 7 22:14:04.336528488 (12h34m3.925873475s) [upload=13057]: Done starting upload Aug 7 22:14:04.336571460 (12h34m3.925916447s) [upload=13057]: Uploading segments... Aug 7 22:14:04.336601788 (12h34m3.925946775s) [upload=13057]: Got next segment. Inline: true Aug 7 22:14:04.336605075 (12h34m3.925950052s) [upload=13057]: Waiting for error group managing segments... Aug 7 22:14:04.336639170 (12h34m3.925984147s) [upload=13057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:04.336643388 (12h34m3.925988375s) [upload=13057]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:14:04.336645732 (12h34m3.925990719s) [upload=13057]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:33.693506183 (12h34m33.28285116s) [upload=13057]: Done uploading segments... Aug 7 22:14:33.693533705 (12h34m33.282878692s) [upload=13057]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:14:56.898136222 (12h34m56.487481239s) [upload=13071]: Starting upload Aug 7 22:14:56.898259367 (12h34m56.487604384s) [upload=13071]: Done starting upload Aug 7 22:14:56.898331565 (12h34m56.487676583s) [upload=13071]: Uploading segments... Aug 7 22:14:56.898390859 (12h34m56.487735866s) [upload=13071]: Got next segment. Inline: true Aug 7 22:14:56.898397631 (12h34m56.487742649s) [upload=13071]: Waiting for error group managing segments... Aug 7 22:14:56.898453538 (12h34m56.487798535s) [upload=13071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:56.898460371 (12h34m56.487805388s) [upload=13071]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:14:56.898467315 (12h34m56.487812332s) [upload=13071]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:15:26.250806715 (12h35m25.840151692s) [upload=13071]: Done uploading segments... Aug 7 22:15:26.250836313 (12h35m25.8401813s) [upload=13071]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:15:34.770168401 (12h35m34.359513388s) [upload=13080]: Starting upload Aug 7 22:15:34.770290034 (12h35m34.359635021s) [upload=13080]: Done starting upload Aug 7 22:15:34.770415243 (12h35m34.35976023s) [upload=13080]: Uploading segments... Aug 7 22:15:34.770513081 (12h35m34.359858068s) [upload=13080]: Got next segment. Inline: true Aug 7 22:15:34.770518170 (12h35m34.359863157s) [upload=13080]: Waiting for error group managing segments... Aug 7 22:15:34.770576111 (12h35m34.359921098s) [upload=13080]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:34.770582192 (12h35m34.359927169s) [upload=13080]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:15:34.770584747 (12h35m34.359929734s) [upload=13080]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:16:04.123598264 (12h36m3.712943251s) [upload=13080]: Done uploading segments... Aug 7 22:16:04.123618482 (12h36m3.712963469s) [upload=13080]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:16:05.006396275 (12h36m4.595741272s) [upload=13084]: Starting upload Aug 7 22:16:05.006480486 (12h36m4.595825473s) [upload=13084]: Done starting upload Aug 7 22:16:05.006520022 (12h36m4.595865009s) [upload=13084]: Uploading segments... Aug 7 22:16:05.006575979 (12h36m4.595920966s) [upload=13084]: Got next segment. Inline: true Aug 7 22:16:05.006582241 (12h36m4.595927228s) [upload=13084]: Waiting for error group managing segments... Aug 7 22:16:05.006619191 (12h36m4.595964178s) [upload=13084]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:16:05.006623339 (12h36m4.595968326s) [upload=13084]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:16:05.006625714 (12h36m4.595970701s) [upload=13084]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:16:34.356940910 (12h36m33.946285897s) [upload=13084]: Done uploading segments... Aug 7 22:16:34.356970046 (12h36m33.946315033s) [upload=13084]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:17:17.980057052 (12h37m17.569402039s) [upload=13103]: Starting upload Aug 7 22:17:17.980153156 (12h37m17.569498144s) [upload=13103]: Done starting upload Aug 7 22:17:17.980255451 (12h37m17.569600429s) [upload=13103]: Uploading segments... Aug 7 22:17:17.980314855 (12h37m17.569659852s) [upload=13103]: Got next segment. Inline: true Aug 7 22:17:17.980321198 (12h37m17.569666205s) [upload=13103]: Waiting for error group managing segments... Aug 7 22:17:17.980417452 (12h37m17.569762439s) [upload=13103]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:17:17.980424405 (12h37m17.569769392s) [upload=13103]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:17:17.980428243 (12h37m17.56977326s) [upload=13103]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:17:47.333713816 (12h37m46.923058803s) [upload=13103]: Done uploading segments... Aug 7 22:17:47.333734415 (12h37m46.923079402s) [upload=13103]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.001418083s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X..X....................X....................X..........] duration: 6.46179794s FAIL (2025-08-07 22:17:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 34e70bda2fe1d7206098dbfb59fc4576ab613f2b752f601da4a5b93473be374e != before ef9ba24509b0003982578e9daec1e6051f8d6ac0bef4f9769b005a8c713045cf ======LOGS====== ding segments... Aug 7 22:10:42.792858699 (12h30m42.382203686s) [upload=12997]: Got next segment. Inline: false Aug 7 22:10:42.792883667 (12h30m42.382228644s) [upload=12997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:10:42.792887915 (12h30m42.382232902s) [upload=12997]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:10:43.239239407 (12h30m42.828584384s) [seg_pos={0 0}, upload=12997]: Begin upload segment... Aug 7 22:10:43.239300844 (12h30m42.828645831s) [seg_pos={0 0}, upload=12997]: Done begin upload segment. Aug 7 22:10:43.239329008 (12h30m42.828673995s) [noise=true , address=104.36.82.61:4394 , piece_id=BZ4OH34HU6COIEBD542MEZ5YGPXPLFMQKIO2EOV23UBTDTFK5XAA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.239541695 (12h30m42.828886692s) [noise=true , address=104.36.82.37:4687 , piece_id=I7ZESSKAS4ZUYX3CUI4ZRZWY2PLRATRNR6FGVGVAXQD6JT3DRUIA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.239923255 (12h30m42.829268242s) [noise=true , address=104.36.82.10:4087 , piece_id=PDZWS5ZZVKC5BCXJIY43NBQVXM7NCLKJZNSVTD3G2K6F2YEY4RVA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.240056509 (12h30m42.829401506s) [noise=true , address=45.91.116.6:4995 , piece_id=GYQIOSTEVRSY6VKX3GQCUC5YZY5S3ZV7R6X2LBG3QRTOOQD53NOA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.240324512 (12h30m42.829669499s) [noise=true , address=104.36.82.38:4705 , piece_id=M557M36RJRMMND4F33G7BOBYVMESQZZN7ZFESJ5CLHEKXWBCL5SA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.240397562 (12h30m42.829742549s) [noise=true , address=79.127.201.215:4351 , piece_id=367MVHSAIIGC6UZQY4EMH25DJCYYTAIUMPFF3PY426RZJHIO44DA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.240579942 (12h30m42.829924929s) [noise=true , address=104.36.82.47:4524 , piece_id=JDRSOKIHNV2RD3COKNJQHRYG2OXOPUBQOG3AHCTYOKKYC3LZISNQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.240814029 (12h30m42.830159016s) [noise=true , address=64.49.13.169:4710 , piece_id=KOUJTCIUNCXFZSJWG6ZKR4KV42WV3ABECSARKOQ5YB5W4LAJTGAA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.240831132 (12h30m42.830176119s) [noise=true , address=104.36.82.59:4331 , piece_id=54OHTXRTR2J4S4HECSB7DFPHSC7T4UGBBC4X7WI72IVCMESI5BQQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.241273939 (12h30m42.830618926s) [noise=true , address=104.36.82.44:4881 , piece_id=YWXSTJSJ46ERQLZ6A2E36CX3R2FI4XVVEZGUMDYJX4ZG3M3R5ACQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.241589863 (12h30m42.83093485s) [noise=true , address=104.36.82.2:4750 , piece_id=24FHN36X2VIWLTYUDHSJNLSOHHEZ56K6DBIGSMI2ZCA26CPT7L7Q, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.241938992 (12h30m42.831283979s) [noise=true , address=104.36.82.3:4791 , piece_id=DJ46BVH37DMTY3YD6VG2AEPNNTPYF7EH4IA3Y633UH3QO6XUTKPQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.242014867 (12h30m42.831359854s) [noise=true , address=104.36.82.58:4270 , piece_id=HLYO6DZ75CNGFXJ6DA5SKQNVPD5COMDHFAETYUZTBOSKMPT4BUEQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.242250207 (12h30m42.831595194s) [noise=true , address=104.36.82.44:4885 , piece_id=FJ726HA3PDRWAYDFGUEK2Y7D3B77ERCFPJSGWNKKPLGWXC3WK7GA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.242484636 (12h30m42.831829623s) [noise=true , address=79.127.205.246:4254 , piece_id=V3AT6KY72VDSIMMOZUC4JECVAQSPAHMAPC5CXDVCO46Z6RTPRUVA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.242691451 (12h30m42.832036438s) [noise=true , address=104.36.82.8:4995 , piece_id=BI77BF2ZGP3ZMB2ABMBUCKQCEF3Z6QHRC7IT6Q2ATIF6VON32WWQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.242894410 (12h30m42.832239397s) [noise=true , address=104.36.82.43:4859 , piece_id=P2S2T5IIUTG4D6EQILFQ7OR5E5JOZHAYO563X7ED5UBN7THKW4QA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.243127415 (12h30m42.832472402s) [noise=true , address=104.36.82.46:4509 , piece_id=WQ3UWQPCMAZ7EDHTSURUQAKL6KWHUA3GSJI6U2G5ZUABRGCTFEXQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:44.351806366 (12h30m43.941151353s) [upload=12997]: Next returned nil segment Aug 7 22:10:44.351819280 (12h30m43.941164257s) [upload=12997]: Waiting for error group managing segments... Aug 7 22:10:44.651888156 (12h30m44.241233144s) [noise=true , address=79.127.205.246:4254 , piece_id=V3AT6KY72VDSIMMOZUC4JECVAQSPAHMAPC5CXDVCO46Z6RTPRUVA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:44.674573650 (12h30m44.263918637s) [noise=true , address=64.49.13.169:4710 , piece_id=KOUJTCIUNCXFZSJWG6ZKR4KV42WV3ABECSARKOQ5YB5W4LAJTGAA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.186610734 (12h30m44.775955721s) [noise=true , address=104.36.82.38:4705 , piece_id=M557M36RJRMMND4F33G7BOBYVMESQZZN7ZFESJ5CLHEKXWBCL5SA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.204653265 (12h30m44.793998252s) [noise=true , address=104.36.82.58:4270 , piece_id=HLYO6DZ75CNGFXJ6DA5SKQNVPD5COMDHFAETYUZTBOSKMPT4BUEQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.218003397 (12h30m44.807348374s) [noise=true , address=79.127.201.215:4351 , piece_id=367MVHSAIIGC6UZQY4EMH25DJCYYTAIUMPFF3PY426RZJHIO44DA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.276599656 (12h30m44.865944643s) [noise=true , address=104.36.82.46:4509 , piece_id=WQ3UWQPCMAZ7EDHTSURUQAKL6KWHUA3GSJI6U2G5ZUABRGCTFEXQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.316254365 (12h30m44.905599352s) [noise=true , address=104.36.82.2:4750 , piece_id=24FHN36X2VIWLTYUDHSJNLSOHHEZ56K6DBIGSMI2ZCA26CPT7L7Q, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.349441480 (12h30m44.938786467s) [noise=true , address=104.36.82.3:4791 , piece_id=DJ46BVH37DMTY3YD6VG2AEPNNTPYF7EH4IA3Y633UH3QO6XUTKPQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.352466728 (12h30m44.941811715s) [noise=true , address=104.36.82.44:4881 , piece_id=YWXSTJSJ46ERQLZ6A2E36CX3R2FI4XVVEZGUMDYJX4ZG3M3R5ACQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.418760245 (12h30m45.008105232s) [noise=true , address=104.36.82.37:4687 , piece_id=I7ZESSKAS4ZUYX3CUI4ZRZWY2PLRATRNR6FGVGVAXQD6JT3DRUIA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.540632266 (12h30m45.129977253s) [noise=true , address=104.36.82.10:4087 , piece_id=PDZWS5ZZVKC5BCXJIY43NBQVXM7NCLKJZNSVTD3G2K6F2YEY4RVA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.567016576 (12h30m45.156361563s) [noise=true , address=104.36.82.59:4331 , piece_id=54OHTXRTR2J4S4HECSB7DFPHSC7T4UGBBC4X7WI72IVCMESI5BQQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.597615539 (12h30m45.186960526s) [noise=true , address=104.36.82.44:4885 , piece_id=FJ726HA3PDRWAYDFGUEK2Y7D3B77ERCFPJSGWNKKPLGWXC3WK7GA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.676238976 (12h30m45.265584013s) [noise=true , address=45.91.116.6:4995 , piece_id=GYQIOSTEVRSY6VKX3GQCUC5YZY5S3ZV7R6X2LBG3QRTOOQD53NOA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.676277709 (12h30m45.265622696s) [seg_pos={0 0}, upload=12997]: Segment reached optimal threshold of 14 pieces. Aug 7 22:10:45.676633781 (12h30m45.265978758s) [noise=true , address=104.36.82.8:4995 , piece_id=BI77BF2ZGP3ZMB2ABMBUCKQCEF3Z6QHRC7IT6Q2ATIF6VON32WWQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tvoTD1LoSj3q2rLiDckfT56dXNKKwdVu1ZAsBZExNFVTom9tH): protocol: expected piece hash; context canceled; EOF Aug 7 22:10:45.676841539 (12h30m45.266186526s) [noise=true , address=104.36.82.43:4859 , piece_id=P2S2T5IIUTG4D6EQILFQ7OR5E5JOZHAYO563X7ED5UBN7THKW4QA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126SqCtRGPziEeopRjA4nNkjkCZWvUwkt5XaMgcqZKJwoxuYWY7): protocol: rpc: dial tcp 104.36.82.43:4859: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:10:45.676869993 (12h30m45.26621498s) [noise=true , address=104.36.82.47:4524 , piece_id=JDRSOKIHNV2RD3COKNJQHRYG2OXOPUBQOG3AHCTYOKKYC3LZISNQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4): protocol: read tcp 188.166.225.170:58322->104.36.82.47:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:10:45.676908757 (12h30m45.266253744s) [noise=true , address=104.36.82.61:4394 , piece_id=BZ4OH34HU6COIEBD542MEZ5YGPXPLFMQKIO2EOV23UBTDTFK5XAA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dSpTqCxeAcL9PVYbP7BhmuKE7papFPc9swB1Fy83WcFDgQbcF): protocol: rpc: dial tcp 104.36.82.61:4394: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:10:45.676927984 (12h30m45.266272961s) [seg_pos={0 0}, upload=12997]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:10:45.677012455 (12h30m45.266357443s) [upload=12997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:10:45.677016543 (12h30m45.26636152s) [upload=12997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:45.998268658 (12h30m45.587613635s) [upload=12997]: Done uploading segments... Aug 7 22:10:45.998285902 (12h30m45.587630889s) [upload=12997]: Upload finished. err: <nil> Duration: 6.461822837s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.......X.................X..................................] duration: 1.980339253s FAIL (2025-08-07 22:17:44 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== 8m27.919686751s) [upload=12612]: Done uploading segments... Aug 7 21:48:28.330358527 (12h8m27.919703514s) [upload=12612]: Upload finished. err: <nil> Aug 7 21:48:54.943937458 (12h8m54.533282445s) [upload=12618]: Starting upload Aug 7 21:48:54.944002622 (12h8m54.533347609s) [upload=12618]: Done starting upload Aug 7 21:48:54.944106100 (12h8m54.533451087s) [upload=12618]: Uploading segments... Aug 7 21:48:54.944196242 (12h8m54.53354132s) [upload=12618]: Got next segment. Inline: true Aug 7 21:48:54.944206543 (12h8m54.53355158s) [upload=12618]: Waiting for error group managing segments... Aug 7 21:48:54.944254484 (12h8m54.533599461s) [upload=12618]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:48:54.944259223 (12h8m54.5336042s) [upload=12618]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:48:54.944262139 (12h8m54.533607126s) [upload=12618]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:48:55.330753187 (12h8m54.920098174s) [upload=12618]: Done uploading segments... Aug 7 21:48:55.330774959 (12h8m54.920119936s) [upload=12618]: Upload finished. err: <nil> Aug 7 21:49:36.650951942 (12h9m36.240296929s) [upload=12631]: Starting upload Aug 7 21:49:36.650993252 (12h9m36.240338239s) [upload=12631]: Done starting upload Aug 7 21:49:36.651049029 (12h9m36.240394016s) [upload=12631]: Uploading segments... Aug 7 21:49:36.651075309 (12h9m36.240420286s) [upload=12631]: Got next segment. Inline: true Aug 7 21:49:36.651078775 (12h9m36.240423763s) [upload=12631]: Waiting for error group managing segments... Aug 7 21:49:36.651115776 (12h9m36.240460773s) [upload=12631]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:49:36.651120254 (12h9m36.240465251s) [upload=12631]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:49:36.651124473 (12h9m36.24046947s) [upload=12631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:49:36.920683881 (12h9m36.510028868s) [upload=12631]: Done uploading segments... Aug 7 21:49:36.920707246 (12h9m36.510052233s) [upload=12631]: Upload finished. err: <nil> Aug 7 21:50:03.363429937 (12h10m2.952774924s) [upload=12636]: Starting upload Aug 7 21:50:03.363493408 (12h10m2.952838405s) [upload=12636]: Done starting upload Aug 7 21:50:03.363582518 (12h10m2.952927515s) [upload=12636]: Uploading segments... Aug 7 21:50:03.363692078 (12h10m2.953037085s) [upload=12636]: Got next segment. Inline: true Aug 7 21:50:03.363699752 (12h10m2.953044739s) [upload=12636]: Waiting for error group managing segments... Aug 7 21:50:03.363764116 (12h10m2.953109103s) [upload=12636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:50:03.363773323 (12h10m2.95311831s) [upload=12636]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:50:03.363777271 (12h10m2.953122258s) [upload=12636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:50:03.636022787 (12h10m3.225367774s) [upload=12636]: Done uploading segments... Aug 7 21:50:03.636042204 (12h10m3.225387202s) [upload=12636]: Upload finished. err: <nil> Aug 7 21:50:20.944550175 (12h10m20.533895152s) [upload=12642]: Starting upload Aug 7 21:50:20.944594911 (12h10m20.533939898s) [upload=12642]: Done starting upload Aug 7 21:50:20.944644405 (12h10m20.533989392s) [upload=12642]: Uploading segments... Aug 7 21:50:20.944674473 (12h10m20.534019461s) [upload=12642]: Got next segment. Inline: true Aug 7 21:50:20.944677820 (12h10m20.534022807s) [upload=12642]: Waiting for error group managing segments... Aug 7 21:50:20.944709450 (12h10m20.534054437s) [upload=12642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:50:20.944715131 (12h10m20.534060118s) [upload=12642]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:50:20.944717505 (12h10m20.534062493s) [upload=12642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:50:21.212077476 (12h10m20.801422473s) [upload=12642]: Done uploading segments... Aug 7 21:50:21.212101602 (12h10m20.801446599s) [upload=12642]: Upload finished. err: <nil> Aug 7 21:51:22.649621444 (12h11m22.238966431s) [upload=12660]: Starting upload Aug 7 21:51:22.649696837 (12h11m22.239041824s) [upload=12660]: Done starting upload Aug 7 21:51:22.649777742 (12h11m22.239122729s) [upload=12660]: Uploading segments... Aug 7 21:51:22.649867044 (12h11m22.239212041s) [upload=12660]: Got next segment. Inline: true Aug 7 21:51:22.649872053 (12h11m22.23921704s) [upload=12660]: Waiting for error group managing segments... Aug 7 21:51:22.649940053 (12h11m22.23928504s) [upload=12660]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:51:22.649946785 (12h11m22.239291772s) [upload=12660]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:51:22.649949932 (12h11m22.239294919s) [upload=12660]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:51:22.914527578 (12h11m22.503872565s) [upload=12660]: Done uploading segments... Aug 7 21:51:22.914549130 (12h11m22.503894117s) [upload=12660]: Upload finished. err: <nil> Aug 7 21:51:38.628060070 (12h11m38.217405057s) [upload=12665]: Starting upload Aug 7 21:51:38.628100548 (12h11m38.217445535s) [upload=12665]: Done starting upload Aug 7 21:51:38.628155373 (12h11m38.21750036s) [upload=12665]: Uploading segments... Aug 7 21:51:38.628187814 (12h11m38.217532802s) [upload=12665]: Got next segment. Inline: true Aug 7 21:51:38.628191381 (12h11m38.217536368s) [upload=12665]: Waiting for error group managing segments... Aug 7 21:51:38.628222691 (12h11m38.217567678s) [upload=12665]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:51:38.628226268 (12h11m38.217571255s) [upload=12665]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:51:38.628228662 (12h11m38.217573649s) [upload=12665]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:51:38.940125222 (12h11m38.529470269s) [upload=12665]: Done uploading segments... Aug 7 21:51:38.940147124 (12h11m38.529492191s) [upload=12665]: Upload finished. err: <nil> Aug 7 21:52:16.262879794 (12h12m15.852224781s) [upload=12676]: Starting upload Aug 7 21:52:16.262939168 (12h12m15.852284155s) [upload=12676]: Done starting upload Aug 7 21:52:16.263047876 (12h12m15.852392873s) [upload=12676]: Uploading segments... Aug 7 21:52:16.263139601 (12h12m15.852484598s) [upload=12676]: Got next segment. Inline: true Aug 7 21:52:16.263144440 (12h12m15.852489417s) [upload=12676]: Waiting for error group managing segments... Aug 7 21:52:16.263191931 (12h12m15.852536918s) [upload=12676]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:52:16.263197472 (12h12m15.852542449s) [upload=12676]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:52:16.263200056 (12h12m15.852545043s) [upload=12676]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:52:16.570506954 (12h12m16.159851931s) [upload=12676]: Done uploading segments... Aug 7 21:52:16.570524978 (12h12m16.159869965s) [upload=12676]: Upload finished. err: <nil> Aug 7 21:53:00.574165558 (12h13m0.163510545s) [upload=12691]: Starting upload Aug 7 21:53:00.574212286 (12h13m0.163557273s) [upload=12691]: Done starting upload Aug 7 21:53:00.574248476 (12h13m0.163593463s) [upload=12691]: Uploading segments... Aug 7 21:53:00.574327307 (12h13m0.163672294s) [upload=12691]: Got next segment. Inline: true Aug 7 21:53:00.574332868 (12h13m0.163677855s) [upload=12691]: Waiting for error group managing segments... Aug 7 21:53:00.574373505 (12h13m0.163718492s) [upload=12691]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:53:00.574390598 (12h13m0.163735585s) [upload=12691]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:53:00.574393373 (12h13m0.16373835s) [upload=12691]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:53:00.840335180 (12h13m0.429680167s) [upload=12691]: Done uploading segments... Aug 7 21:53:00.840351500 (12h13m0.429696487s) [upload=12691]: Upload finished. err: <nil> Aug 7 21:53:46.136340579 (12h13m45.725685566s) [upload=12705]: Starting upload Aug 7 21:53:46.136382029 (12h13m45.725727016s) [upload=12705]: Done starting upload Aug 7 21:53:46.136495205 (12h13m45.725840192s) [upload=12705]: Uploading segments... Aug 7 21:53:46.136529601 (12h13m45.725874588s) [upload=12705]: Got next segment. Inline: true Aug 7 21:53:46.136533228 (12h13m45.725878215s) [upload=12705]: Waiting for error group managing segments... Aug 7 21:53:46.136581701 (12h13m45.725926688s) [upload=12705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:53:46.136585949 (12h13m45.725930936s) [upload=12705]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:53:46.136588393 (12h13m45.72593338s) [upload=12705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:53:46.427021148 (12h13m46.016366155s) [upload=12705]: Done uploading segments... Aug 7 21:53:46.427045194 (12h13m46.016390181s) [upload=12705]: Upload finished. err: <nil> Aug 7 21:54:41.670591536 (12h14m41.259936523s) [upload=12721]: Starting upload Aug 7 21:54:41.670632916 (12h14m41.259977903s) [upload=12721]: Done starting upload Aug 7 21:54:41.670677612 (12h14m41.260022599s) [upload=12721]: Uploading segments... Aug 7 21:54:41.670708571 (12h14m41.260053558s) [upload=12721]: Got next segment. Inline: true Aug 7 21:54:41.670713510 (12h14m41.260058497s) [upload=12721]: Waiting for error group managing segments... Aug 7 21:54:41.670746032 (12h14m41.260091019s) [upload=12721]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:54:41.670749979 (12h14m41.260094966s) [upload=12721]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:54:41.670752424 (12h14m41.260097411s) [upload=12721]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:54:41.940759391 (12h14m41.530104368s) [upload=12721]: Done uploading segments... Aug 7 21:54:41.940776734 (12h14m41.530121721s) [upload=12721]: Upload finished. err: <nil> Duration: 1.980365823s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXX.X.X.X................................XXXXXXX] duration: 30.000682381s FAIL (2025-08-07 22:17:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 55.438319623 (12h30m55.0276646s) [noise=true , address=45.91.116.5:4298 , piece_id=XJIIUMO7PI6XTBLXE7IRPS2YYI53MRGPEW7HS5CWFXUSIXBT5MBQ, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.440502008 (12h30m55.029846995s) [noise=true , address=45.91.116.13:4154 , piece_id=TPFKH6V3LAM6B2O7L744BZPLT64I2BZVYJXFPEPLIGMVEQ7246YQ, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.449485475 (12h30m55.038830462s) [noise=true , address=104.36.82.7:4909 , piece_id=6ZPKL2IEZN54MOXLAC7P4GURLWMJ7MUBNCBXPUGF6TD6257G7TAA, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.449773497 (12h30m55.039118484s) [noise=true , address=77.90.31.13:4627 , piece_id=3IQP6QL756UL7CNL357KLNUTHQCSEG5337GUT2PD2AS623PTIYCA, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.486432234 (12h30m55.075777221s) [noise=true , address=77.90.31.11:4555 , piece_id=A6WWFITSRKE4GVTQODOYU36ORK7OVE3MVVR7B4FPSSP64DATOQ6A, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.522908903 (12h30m55.11225389s) [noise=true , address=77.90.31.18:4805 , piece_id=VQ3CGKCPEKYPCAIOHQ2FZSINJBHBCCQUZD2AD4AIHX72SZN7SEKA, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.546118308 (12h30m55.135463295s) [noise=true , address=104.36.82.4:4802 , piece_id=B4Q65J747GHE2ZRZDM7RLWC3AQC7TLK5D53REAUA5DMHT3TGA44Q, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.568801126 (12h30m55.158146113s) [noise=true , address=104.36.82.4:4804 , piece_id=BI6ZPR56BKHTA2QGM3BNUBFJM6PAD4K27P5TYHRGLG4NBD56LC4A, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.587208293 (12h30m55.176553341s) [noise=true , address=77.90.31.17:4770 , piece_id=BG2ZL7XC52NEJC5FRS4V75MEPJH34B27IPLTJNQLSX4N3OZGVHXA, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.587245686 (12h30m55.176590713s) [seg_pos={0 0}, upload=13000]: Segment reached optimal threshold of 13 pieces. Aug 7 22:10:55.587371516 (12h30m55.176716533s) [noise=true , address=45.91.116.4:4939 , piece_id=FQ4Z6Y4KIOGJLPCZFVPCUJSYOMGDOTZNAIEKGNWKASCLKOWLCTUQ, seg_pos={0 0}, upload=13000]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Aug 7 22:10:55.587442001 (12h30m55.176786988s) [noise=true , address=45.91.116.7:4009 , piece_id=47X2WBQ4XYRNAZDBVU572QSRHXUNSYRCOUALM4SF4WRHCDW6QTCA, seg_pos={0 0}, upload=13000]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Aug 7 22:10:55.587644678 (12h30m55.176989726s) [noise=true , address=104.36.82.12:4150 , piece_id=XG7E3FHDR2VCPREGTWFFJ52JLGUPRUKQJ75KFYEUXPI4OUNTB3ZA, seg_pos={0 0}, upload=13000]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: context canceled; context canceled Aug 7 22:10:55.587681550 (12h30m55.177026557s) [seg_pos={0 0}, upload=13000]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 22:10:55.587777072 (12h30m55.177122109s) [upload=13000]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:10:55.587787912 (12h30m55.177132929s) [upload=13000]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:55.887211865 (12h30m55.476556883s) [upload=13000]: Done uploading segments... Aug 7 22:10:55.887228857 (12h30m55.476573844s) [upload=13000]: Upload finished. err: <nil> Aug 7 22:11:40.872949328 (12h31m40.462294315s) [upload=13014]: Starting upload Aug 7 22:11:40.873003161 (12h31m40.462348148s) [upload=13014]: Done starting upload Aug 7 22:11:40.873072924 (12h31m40.462417911s) [upload=13014]: Uploading segments... Aug 7 22:11:40.873431210 (12h31m40.462776287s) [upload=13014]: Got next segment. Inline: false Aug 7 22:11:40.873471537 (12h31m40.462816525s) [upload=13014]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:11:40.873480314 (12h31m40.462825331s) [upload=13014]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:12:10.229450117 (12h32m9.818795124s) [upload=13014]: Next segment err: context deadline exceeded Aug 7 22:12:10.229589654 (12h32m9.818934631s) [upload=13014]: Done uploading segments... Aug 7 22:12:10.229603972 (12h32m9.818948949s) [upload=13014]: Upload finished. err: context deadline exceeded Aug 7 22:12:48.993762773 (12h32m48.58310776s) [upload=13033]: Starting upload Aug 7 22:12:48.993844269 (12h32m48.583189246s) [upload=13033]: Done starting upload Aug 7 22:12:48.993904404 (12h32m48.583249391s) [upload=13033]: Uploading segments... Aug 7 22:12:48.994022439 (12h32m48.583367427s) [upload=13033]: Got next segment. Inline: false Aug 7 22:12:48.994048309 (12h32m48.583393296s) [upload=13033]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:48.994052647 (12h32m48.583397634s) [upload=13033]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:13:18.347381136 (12h33m17.936726133s) [upload=13033]: Next segment err: context deadline exceeded Aug 7 22:13:18.347514681 (12h33m17.936859668s) [upload=13033]: Done uploading segments... Aug 7 22:13:18.347529670 (12h33m17.936874657s) [upload=13033]: Upload finished. err: context deadline exceeded Aug 7 22:13:51.696413792 (12h33m51.285758769s) [upload=13052]: Starting upload Aug 7 22:13:51.696517580 (12h33m51.285862567s) [upload=13052]: Done starting upload Aug 7 22:13:51.696601842 (12h33m51.285946829s) [upload=13052]: Uploading segments... Aug 7 22:13:51.696787407 (12h33m51.286132414s) [upload=13052]: Got next segment. Inline: false Aug 7 22:13:51.696823415 (12h33m51.286168413s) [upload=13052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:13:51.696829066 (12h33m51.286174073s) [upload=13052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:14:21.042712175 (12h34m20.632057152s) [upload=13052]: Next segment err: context deadline exceeded Aug 7 22:14:21.042793340 (12h34m20.632138327s) [upload=13052]: Done uploading segments... Aug 7 22:14:21.042809000 (12h34m20.632153987s) [upload=13052]: Upload finished. err: context deadline exceeded Aug 7 22:14:36.252577987 (12h34m35.841922974s) [upload=13065]: Starting upload Aug 7 22:14:36.252631760 (12h34m35.841976747s) [upload=13065]: Done starting upload Aug 7 22:14:36.252692656 (12h34m35.842037643s) [upload=13065]: Uploading segments... Aug 7 22:14:36.252805532 (12h34m35.842150519s) [upload=13065]: Got next segment. Inline: false Aug 7 22:14:36.252856870 (12h34m35.842201857s) [upload=13065]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:36.252863202 (12h34m35.842208189s) [upload=13065]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:15:05.607209972 (12h35m5.196554959s) [upload=13065]: Next segment err: context deadline exceeded Aug 7 22:15:05.607234078 (12h35m5.196579065s) [upload=13065]: Done uploading segments... Aug 7 22:15:05.607262002 (12h35m5.196606989s) [upload=13065]: Upload finished. err: context deadline exceeded Aug 7 22:15:30.794674976 (12h35m30.384019963s) [upload=13077]: Starting upload Aug 7 22:15:30.794787000 (12h35m30.384132008s) [upload=13077]: Done starting upload Aug 7 22:15:30.794881030 (12h35m30.384226007s) [upload=13077]: Uploading segments... Aug 7 22:15:30.795138724 (12h35m30.384483711s) [upload=13077]: Got next segment. Inline: false Aug 7 22:15:30.795218847 (12h35m30.384563834s) [upload=13077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:30.795225099 (12h35m30.384570106s) [upload=13077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:16:00.147601500 (12h35m59.736946487s) [upload=13077]: Next segment err: context deadline exceeded Aug 7 22:16:00.147674640 (12h35m59.737019628s) [upload=13077]: Done uploading segments... Aug 7 22:16:00.147688897 (12h35m59.737033884s) [upload=13077]: Upload finished. err: context deadline exceeded Aug 7 22:16:34.181460693 (12h36m33.77080568s) [upload=13093]: Starting upload Aug 7 22:16:34.181519996 (12h36m33.770864983s) [upload=13093]: Done starting upload Aug 7 22:16:34.181628824 (12h36m33.770973811s) [upload=13093]: Uploading segments... Aug 7 22:16:34.181727263 (12h36m33.77107226s) [upload=13093]: Got next segment. Inline: false Aug 7 22:16:34.181766387 (12h36m33.771111385s) [upload=13093]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:16:34.181772269 (12h36m33.771117267s) [upload=13093]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:17:03.528495677 (12h37m3.117840664s) [upload=13093]: Next segment err: context deadline exceeded Aug 7 22:17:03.528680499 (12h37m3.118025497s) [upload=13093]: Done uploading segments... Aug 7 22:17:03.528713574 (12h37m3.118058551s) [upload=13093]: Upload finished. err: context deadline exceeded Aug 7 22:17:08.817549216 (12h37m8.406894193s) [upload=13098]: Starting upload Aug 7 22:17:08.817602037 (12h37m8.406947024s) [upload=13098]: Done starting upload Aug 7 22:17:08.817716647 (12h37m8.407061634s) [upload=13098]: Uploading segments... Aug 7 22:17:08.817823451 (12h37m8.407168428s) [upload=13098]: Got next segment. Inline: false Aug 7 22:17:08.817862286 (12h37m8.407207273s) [upload=13098]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:17:08.817866333 (12h37m8.407211311s) [upload=13098]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:17:38.170422707 (12h37m37.759767684s) [upload=13098]: Next segment err: context deadline exceeded Aug 7 22:17:38.170520234 (12h37m37.759865211s) [upload=13098]: Done uploading segments... Aug 7 22:17:38.170546113 (12h37m37.7598911s) [upload=13098]: Upload finished. err: context deadline exceeded Duration: 30.000700456s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 25 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXX...X................................XXXXXXXXXX] duration: 30.001172068s FAIL (2025-08-07 22:17:38 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.001192557s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [........X.................X.................................] duration: 1.904980823s FAIL (2025-08-07 22:17:29 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== 8m27.919686751s) [upload=12612]: Done uploading segments... Aug 7 21:48:28.330358527 (12h8m27.919703514s) [upload=12612]: Upload finished. err: <nil> Aug 7 21:48:54.943937458 (12h8m54.533282445s) [upload=12618]: Starting upload Aug 7 21:48:54.944002622 (12h8m54.533347609s) [upload=12618]: Done starting upload Aug 7 21:48:54.944106100 (12h8m54.533451087s) [upload=12618]: Uploading segments... Aug 7 21:48:54.944196242 (12h8m54.53354132s) [upload=12618]: Got next segment. Inline: true Aug 7 21:48:54.944206543 (12h8m54.53355158s) [upload=12618]: Waiting for error group managing segments... Aug 7 21:48:54.944254484 (12h8m54.533599461s) [upload=12618]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:48:54.944259223 (12h8m54.5336042s) [upload=12618]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:48:54.944262139 (12h8m54.533607126s) [upload=12618]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:48:55.330753187 (12h8m54.920098174s) [upload=12618]: Done uploading segments... Aug 7 21:48:55.330774959 (12h8m54.920119936s) [upload=12618]: Upload finished. err: <nil> Aug 7 21:49:36.650951942 (12h9m36.240296929s) [upload=12631]: Starting upload Aug 7 21:49:36.650993252 (12h9m36.240338239s) [upload=12631]: Done starting upload Aug 7 21:49:36.651049029 (12h9m36.240394016s) [upload=12631]: Uploading segments... Aug 7 21:49:36.651075309 (12h9m36.240420286s) [upload=12631]: Got next segment. Inline: true Aug 7 21:49:36.651078775 (12h9m36.240423763s) [upload=12631]: Waiting for error group managing segments... Aug 7 21:49:36.651115776 (12h9m36.240460773s) [upload=12631]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:49:36.651120254 (12h9m36.240465251s) [upload=12631]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:49:36.651124473 (12h9m36.24046947s) [upload=12631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:49:36.920683881 (12h9m36.510028868s) [upload=12631]: Done uploading segments... Aug 7 21:49:36.920707246 (12h9m36.510052233s) [upload=12631]: Upload finished. err: <nil> Aug 7 21:50:03.363429937 (12h10m2.952774924s) [upload=12636]: Starting upload Aug 7 21:50:03.363493408 (12h10m2.952838405s) [upload=12636]: Done starting upload Aug 7 21:50:03.363582518 (12h10m2.952927515s) [upload=12636]: Uploading segments... Aug 7 21:50:03.363692078 (12h10m2.953037085s) [upload=12636]: Got next segment. Inline: true Aug 7 21:50:03.363699752 (12h10m2.953044739s) [upload=12636]: Waiting for error group managing segments... Aug 7 21:50:03.363764116 (12h10m2.953109103s) [upload=12636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:50:03.363773323 (12h10m2.95311831s) [upload=12636]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:50:03.363777271 (12h10m2.953122258s) [upload=12636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:50:03.636022787 (12h10m3.225367774s) [upload=12636]: Done uploading segments... Aug 7 21:50:03.636042204 (12h10m3.225387202s) [upload=12636]: Upload finished. err: <nil> Aug 7 21:50:20.944550175 (12h10m20.533895152s) [upload=12642]: Starting upload Aug 7 21:50:20.944594911 (12h10m20.533939898s) [upload=12642]: Done starting upload Aug 7 21:50:20.944644405 (12h10m20.533989392s) [upload=12642]: Uploading segments... Aug 7 21:50:20.944674473 (12h10m20.534019461s) [upload=12642]: Got next segment. Inline: true Aug 7 21:50:20.944677820 (12h10m20.534022807s) [upload=12642]: Waiting for error group managing segments... Aug 7 21:50:20.944709450 (12h10m20.534054437s) [upload=12642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:50:20.944715131 (12h10m20.534060118s) [upload=12642]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:50:20.944717505 (12h10m20.534062493s) [upload=12642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:50:21.212077476 (12h10m20.801422473s) [upload=12642]: Done uploading segments... Aug 7 21:50:21.212101602 (12h10m20.801446599s) [upload=12642]: Upload finished. err: <nil> Aug 7 21:51:22.649621444 (12h11m22.238966431s) [upload=12660]: Starting upload Aug 7 21:51:22.649696837 (12h11m22.239041824s) [upload=12660]: Done starting upload Aug 7 21:51:22.649777742 (12h11m22.239122729s) [upload=12660]: Uploading segments... Aug 7 21:51:22.649867044 (12h11m22.239212041s) [upload=12660]: Got next segment. Inline: true Aug 7 21:51:22.649872053 (12h11m22.23921704s) [upload=12660]: Waiting for error group managing segments... Aug 7 21:51:22.649940053 (12h11m22.23928504s) [upload=12660]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:51:22.649946785 (12h11m22.239291772s) [upload=12660]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:51:22.649949932 (12h11m22.239294919s) [upload=12660]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:51:22.914527578 (12h11m22.503872565s) [upload=12660]: Done uploading segments... Aug 7 21:51:22.914549130 (12h11m22.503894117s) [upload=12660]: Upload finished. err: <nil> Aug 7 21:51:38.628060070 (12h11m38.217405057s) [upload=12665]: Starting upload Aug 7 21:51:38.628100548 (12h11m38.217445535s) [upload=12665]: Done starting upload Aug 7 21:51:38.628155373 (12h11m38.21750036s) [upload=12665]: Uploading segments... Aug 7 21:51:38.628187814 (12h11m38.217532802s) [upload=12665]: Got next segment. Inline: true Aug 7 21:51:38.628191381 (12h11m38.217536368s) [upload=12665]: Waiting for error group managing segments... Aug 7 21:51:38.628222691 (12h11m38.217567678s) [upload=12665]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:51:38.628226268 (12h11m38.217571255s) [upload=12665]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:51:38.628228662 (12h11m38.217573649s) [upload=12665]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:51:38.940125222 (12h11m38.529470269s) [upload=12665]: Done uploading segments... Aug 7 21:51:38.940147124 (12h11m38.529492191s) [upload=12665]: Upload finished. err: <nil> Aug 7 21:52:16.262879794 (12h12m15.852224781s) [upload=12676]: Starting upload Aug 7 21:52:16.262939168 (12h12m15.852284155s) [upload=12676]: Done starting upload Aug 7 21:52:16.263047876 (12h12m15.852392873s) [upload=12676]: Uploading segments... Aug 7 21:52:16.263139601 (12h12m15.852484598s) [upload=12676]: Got next segment. Inline: true Aug 7 21:52:16.263144440 (12h12m15.852489417s) [upload=12676]: Waiting for error group managing segments... Aug 7 21:52:16.263191931 (12h12m15.852536918s) [upload=12676]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:52:16.263197472 (12h12m15.852542449s) [upload=12676]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:52:16.263200056 (12h12m15.852545043s) [upload=12676]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:52:16.570506954 (12h12m16.159851931s) [upload=12676]: Done uploading segments... Aug 7 21:52:16.570524978 (12h12m16.159869965s) [upload=12676]: Upload finished. err: <nil> Aug 7 21:53:00.574165558 (12h13m0.163510545s) [upload=12691]: Starting upload Aug 7 21:53:00.574212286 (12h13m0.163557273s) [upload=12691]: Done starting upload Aug 7 21:53:00.574248476 (12h13m0.163593463s) [upload=12691]: Uploading segments... Aug 7 21:53:00.574327307 (12h13m0.163672294s) [upload=12691]: Got next segment. Inline: true Aug 7 21:53:00.574332868 (12h13m0.163677855s) [upload=12691]: Waiting for error group managing segments... Aug 7 21:53:00.574373505 (12h13m0.163718492s) [upload=12691]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:53:00.574390598 (12h13m0.163735585s) [upload=12691]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:53:00.574393373 (12h13m0.16373835s) [upload=12691]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:53:00.840335180 (12h13m0.429680167s) [upload=12691]: Done uploading segments... Aug 7 21:53:00.840351500 (12h13m0.429696487s) [upload=12691]: Upload finished. err: <nil> Aug 7 21:53:46.136340579 (12h13m45.725685566s) [upload=12705]: Starting upload Aug 7 21:53:46.136382029 (12h13m45.725727016s) [upload=12705]: Done starting upload Aug 7 21:53:46.136495205 (12h13m45.725840192s) [upload=12705]: Uploading segments... Aug 7 21:53:46.136529601 (12h13m45.725874588s) [upload=12705]: Got next segment. Inline: true Aug 7 21:53:46.136533228 (12h13m45.725878215s) [upload=12705]: Waiting for error group managing segments... Aug 7 21:53:46.136581701 (12h13m45.725926688s) [upload=12705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:53:46.136585949 (12h13m45.725930936s) [upload=12705]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:53:46.136588393 (12h13m45.72593338s) [upload=12705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:53:46.427021148 (12h13m46.016366155s) [upload=12705]: Done uploading segments... Aug 7 21:53:46.427045194 (12h13m46.016390181s) [upload=12705]: Upload finished. err: <nil> Aug 7 21:54:41.670591536 (12h14m41.259936523s) [upload=12721]: Starting upload Aug 7 21:54:41.670632916 (12h14m41.259977903s) [upload=12721]: Done starting upload Aug 7 21:54:41.670677612 (12h14m41.260022599s) [upload=12721]: Uploading segments... Aug 7 21:54:41.670708571 (12h14m41.260053558s) [upload=12721]: Got next segment. Inline: true Aug 7 21:54:41.670713510 (12h14m41.260058497s) [upload=12721]: Waiting for error group managing segments... Aug 7 21:54:41.670746032 (12h14m41.260091019s) [upload=12721]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:54:41.670749979 (12h14m41.260094966s) [upload=12721]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:54:41.670752424 (12h14m41.260097411s) [upload=12721]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:54:41.940759391 (12h14m41.530104368s) [upload=12721]: Done uploading segments... Aug 7 21:54:41.940776734 (12h14m41.530121721s) [upload=12721]: Upload finished. err: <nil> Duration: 1.905003877s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [XXXXXXXXXXXX.......X................................XXXXXXXX] duration: 30.000423983s FAIL (2025-08-07 22:17:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000434113s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....X..X....................X....................X.........] duration: 7.642968098s FAIL (2025-08-07 22:17:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 34e70bda2fe1d7206098dbfb59fc4576ab613f2b752f601da4a5b93473be374e != before ef9ba24509b0003982578e9daec1e6051f8d6ac0bef4f9769b005a8c713045cf ======LOGS====== ding segments... Aug 7 22:10:42.792858699 (12h30m42.382203686s) [upload=12997]: Got next segment. Inline: false Aug 7 22:10:42.792883667 (12h30m42.382228644s) [upload=12997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:10:42.792887915 (12h30m42.382232902s) [upload=12997]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:10:43.239239407 (12h30m42.828584384s) [seg_pos={0 0}, upload=12997]: Begin upload segment... Aug 7 22:10:43.239300844 (12h30m42.828645831s) [seg_pos={0 0}, upload=12997]: Done begin upload segment. Aug 7 22:10:43.239329008 (12h30m42.828673995s) [noise=true , address=104.36.82.61:4394 , piece_id=BZ4OH34HU6COIEBD542MEZ5YGPXPLFMQKIO2EOV23UBTDTFK5XAA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.239541695 (12h30m42.828886692s) [noise=true , address=104.36.82.37:4687 , piece_id=I7ZESSKAS4ZUYX3CUI4ZRZWY2PLRATRNR6FGVGVAXQD6JT3DRUIA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.239923255 (12h30m42.829268242s) [noise=true , address=104.36.82.10:4087 , piece_id=PDZWS5ZZVKC5BCXJIY43NBQVXM7NCLKJZNSVTD3G2K6F2YEY4RVA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.240056509 (12h30m42.829401506s) [noise=true , address=45.91.116.6:4995 , piece_id=GYQIOSTEVRSY6VKX3GQCUC5YZY5S3ZV7R6X2LBG3QRTOOQD53NOA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.240324512 (12h30m42.829669499s) [noise=true , address=104.36.82.38:4705 , piece_id=M557M36RJRMMND4F33G7BOBYVMESQZZN7ZFESJ5CLHEKXWBCL5SA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.240397562 (12h30m42.829742549s) [noise=true , address=79.127.201.215:4351 , piece_id=367MVHSAIIGC6UZQY4EMH25DJCYYTAIUMPFF3PY426RZJHIO44DA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.240579942 (12h30m42.829924929s) [noise=true , address=104.36.82.47:4524 , piece_id=JDRSOKIHNV2RD3COKNJQHRYG2OXOPUBQOG3AHCTYOKKYC3LZISNQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.240814029 (12h30m42.830159016s) [noise=true , address=64.49.13.169:4710 , piece_id=KOUJTCIUNCXFZSJWG6ZKR4KV42WV3ABECSARKOQ5YB5W4LAJTGAA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.240831132 (12h30m42.830176119s) [noise=true , address=104.36.82.59:4331 , piece_id=54OHTXRTR2J4S4HECSB7DFPHSC7T4UGBBC4X7WI72IVCMESI5BQQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.241273939 (12h30m42.830618926s) [noise=true , address=104.36.82.44:4881 , piece_id=YWXSTJSJ46ERQLZ6A2E36CX3R2FI4XVVEZGUMDYJX4ZG3M3R5ACQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.241589863 (12h30m42.83093485s) [noise=true , address=104.36.82.2:4750 , piece_id=24FHN36X2VIWLTYUDHSJNLSOHHEZ56K6DBIGSMI2ZCA26CPT7L7Q, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.241938992 (12h30m42.831283979s) [noise=true , address=104.36.82.3:4791 , piece_id=DJ46BVH37DMTY3YD6VG2AEPNNTPYF7EH4IA3Y633UH3QO6XUTKPQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.242014867 (12h30m42.831359854s) [noise=true , address=104.36.82.58:4270 , piece_id=HLYO6DZ75CNGFXJ6DA5SKQNVPD5COMDHFAETYUZTBOSKMPT4BUEQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.242250207 (12h30m42.831595194s) [noise=true , address=104.36.82.44:4885 , piece_id=FJ726HA3PDRWAYDFGUEK2Y7D3B77ERCFPJSGWNKKPLGWXC3WK7GA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.242484636 (12h30m42.831829623s) [noise=true , address=79.127.205.246:4254 , piece_id=V3AT6KY72VDSIMMOZUC4JECVAQSPAHMAPC5CXDVCO46Z6RTPRUVA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.242691451 (12h30m42.832036438s) [noise=true , address=104.36.82.8:4995 , piece_id=BI77BF2ZGP3ZMB2ABMBUCKQCEF3Z6QHRC7IT6Q2ATIF6VON32WWQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.242894410 (12h30m42.832239397s) [noise=true , address=104.36.82.43:4859 , piece_id=P2S2T5IIUTG4D6EQILFQ7OR5E5JOZHAYO563X7ED5UBN7THKW4QA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.243127415 (12h30m42.832472402s) [noise=true , address=104.36.82.46:4509 , piece_id=WQ3UWQPCMAZ7EDHTSURUQAKL6KWHUA3GSJI6U2G5ZUABRGCTFEXQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:44.351806366 (12h30m43.941151353s) [upload=12997]: Next returned nil segment Aug 7 22:10:44.351819280 (12h30m43.941164257s) [upload=12997]: Waiting for error group managing segments... Aug 7 22:10:44.651888156 (12h30m44.241233144s) [noise=true , address=79.127.205.246:4254 , piece_id=V3AT6KY72VDSIMMOZUC4JECVAQSPAHMAPC5CXDVCO46Z6RTPRUVA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:44.674573650 (12h30m44.263918637s) [noise=true , address=64.49.13.169:4710 , piece_id=KOUJTCIUNCXFZSJWG6ZKR4KV42WV3ABECSARKOQ5YB5W4LAJTGAA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.186610734 (12h30m44.775955721s) [noise=true , address=104.36.82.38:4705 , piece_id=M557M36RJRMMND4F33G7BOBYVMESQZZN7ZFESJ5CLHEKXWBCL5SA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.204653265 (12h30m44.793998252s) [noise=true , address=104.36.82.58:4270 , piece_id=HLYO6DZ75CNGFXJ6DA5SKQNVPD5COMDHFAETYUZTBOSKMPT4BUEQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.218003397 (12h30m44.807348374s) [noise=true , address=79.127.201.215:4351 , piece_id=367MVHSAIIGC6UZQY4EMH25DJCYYTAIUMPFF3PY426RZJHIO44DA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.276599656 (12h30m44.865944643s) [noise=true , address=104.36.82.46:4509 , piece_id=WQ3UWQPCMAZ7EDHTSURUQAKL6KWHUA3GSJI6U2G5ZUABRGCTFEXQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.316254365 (12h30m44.905599352s) [noise=true , address=104.36.82.2:4750 , piece_id=24FHN36X2VIWLTYUDHSJNLSOHHEZ56K6DBIGSMI2ZCA26CPT7L7Q, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.349441480 (12h30m44.938786467s) [noise=true , address=104.36.82.3:4791 , piece_id=DJ46BVH37DMTY3YD6VG2AEPNNTPYF7EH4IA3Y633UH3QO6XUTKPQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.352466728 (12h30m44.941811715s) [noise=true , address=104.36.82.44:4881 , piece_id=YWXSTJSJ46ERQLZ6A2E36CX3R2FI4XVVEZGUMDYJX4ZG3M3R5ACQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.418760245 (12h30m45.008105232s) [noise=true , address=104.36.82.37:4687 , piece_id=I7ZESSKAS4ZUYX3CUI4ZRZWY2PLRATRNR6FGVGVAXQD6JT3DRUIA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.540632266 (12h30m45.129977253s) [noise=true , address=104.36.82.10:4087 , piece_id=PDZWS5ZZVKC5BCXJIY43NBQVXM7NCLKJZNSVTD3G2K6F2YEY4RVA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.567016576 (12h30m45.156361563s) [noise=true , address=104.36.82.59:4331 , piece_id=54OHTXRTR2J4S4HECSB7DFPHSC7T4UGBBC4X7WI72IVCMESI5BQQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.597615539 (12h30m45.186960526s) [noise=true , address=104.36.82.44:4885 , piece_id=FJ726HA3PDRWAYDFGUEK2Y7D3B77ERCFPJSGWNKKPLGWXC3WK7GA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.676238976 (12h30m45.265584013s) [noise=true , address=45.91.116.6:4995 , piece_id=GYQIOSTEVRSY6VKX3GQCUC5YZY5S3ZV7R6X2LBG3QRTOOQD53NOA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.676277709 (12h30m45.265622696s) [seg_pos={0 0}, upload=12997]: Segment reached optimal threshold of 14 pieces. Aug 7 22:10:45.676633781 (12h30m45.265978758s) [noise=true , address=104.36.82.8:4995 , piece_id=BI77BF2ZGP3ZMB2ABMBUCKQCEF3Z6QHRC7IT6Q2ATIF6VON32WWQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tvoTD1LoSj3q2rLiDckfT56dXNKKwdVu1ZAsBZExNFVTom9tH): protocol: expected piece hash; context canceled; EOF Aug 7 22:10:45.676841539 (12h30m45.266186526s) [noise=true , address=104.36.82.43:4859 , piece_id=P2S2T5IIUTG4D6EQILFQ7OR5E5JOZHAYO563X7ED5UBN7THKW4QA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126SqCtRGPziEeopRjA4nNkjkCZWvUwkt5XaMgcqZKJwoxuYWY7): protocol: rpc: dial tcp 104.36.82.43:4859: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:10:45.676869993 (12h30m45.26621498s) [noise=true , address=104.36.82.47:4524 , piece_id=JDRSOKIHNV2RD3COKNJQHRYG2OXOPUBQOG3AHCTYOKKYC3LZISNQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4): protocol: read tcp 188.166.225.170:58322->104.36.82.47:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:10:45.676908757 (12h30m45.266253744s) [noise=true , address=104.36.82.61:4394 , piece_id=BZ4OH34HU6COIEBD542MEZ5YGPXPLFMQKIO2EOV23UBTDTFK5XAA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dSpTqCxeAcL9PVYbP7BhmuKE7papFPc9swB1Fy83WcFDgQbcF): protocol: rpc: dial tcp 104.36.82.61:4394: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:10:45.676927984 (12h30m45.266272961s) [seg_pos={0 0}, upload=12997]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:10:45.677012455 (12h30m45.266357443s) [upload=12997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:10:45.677016543 (12h30m45.26636152s) [upload=12997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:45.998268658 (12h30m45.587613635s) [upload=12997]: Done uploading segments... Aug 7 22:10:45.998285902 (12h30m45.587630889s) [upload=12997]: Upload finished. err: <nil> Duration: 7.642996872s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 19 out of 60 is failed (threshold=3) [XXXXXXXXXX.X........................................XXXXXXXX] duration: 30.000582415s FAIL (2025-08-07 22:17:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.00060102s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 27 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXX...X..X............................XXXXXX] duration: 1m0.001106806s FAIL (2025-08-07 22:17:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001131543s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 26 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXX.XX..X...............................XXXXXX] duration: 1m0.000865564s FAIL (2025-08-07 22:17:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Starting upload Aug 7 22:08:12.564392563 (12h28m12.15373757s) [upload=12951]: Done starting upload Aug 7 22:08:12.564478317 (12h28m12.153823314s) [upload=12951]: Uploading segments... Aug 7 22:08:12.564532671 (12h28m12.153877658s) [upload=12951]: Got next segment. Inline: true Aug 7 22:08:12.564537029 (12h28m12.153882006s) [upload=12951]: Waiting for error group managing segments... Aug 7 22:08:12.564572748 (12h28m12.153917735s) [upload=12951]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:08:12.564577227 (12h28m12.153922204s) [upload=12951]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:08:12.564579511 (12h28m12.153924498s) [upload=12951]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:08:12.853863397 (12h28m12.443208384s) [upload=12951]: Done uploading segments... Aug 7 22:08:12.853882363 (12h28m12.44322735s) [upload=12951]: Upload finished. err: <nil> Aug 7 22:08:48.226332068 (12h28m47.815677055s) [upload=12963]: Starting upload Aug 7 22:08:48.226393765 (12h28m47.815738743s) [upload=12963]: Done starting upload Aug 7 22:08:48.226441998 (12h28m47.815786975s) [upload=12963]: Uploading segments... Aug 7 22:08:48.226471044 (12h28m47.815816031s) [upload=12963]: Got next segment. Inline: true Aug 7 22:08:48.226474330 (12h28m47.815819317s) [upload=12963]: Waiting for error group managing segments... Aug 7 22:08:48.226505239 (12h28m47.815850216s) [upload=12963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:08:48.226508785 (12h28m47.815853772s) [upload=12963]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:08:48.226511250 (12h28m47.815856237s) [upload=12963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:08:48.493962093 (12h28m48.08330708s) [upload=12963]: Done uploading segments... Aug 7 22:08:48.493980028 (12h28m48.083325015s) [upload=12963]: Upload finished. err: <nil> Aug 7 22:09:16.735671384 (12h29m16.325016371s) [upload=12973]: Starting upload Aug 7 22:09:16.735728593 (12h29m16.32507358s) [upload=12973]: Done starting upload Aug 7 22:09:16.735770444 (12h29m16.325115431s) [upload=12973]: Uploading segments... Aug 7 22:09:16.735799409 (12h29m16.325144396s) [upload=12973]: Got next segment. Inline: true Aug 7 22:09:16.735802535 (12h29m16.325147522s) [upload=12973]: Waiting for error group managing segments... Aug 7 22:09:16.735831881 (12h29m16.325176868s) [upload=12973]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:09:16.735835217 (12h29m16.325180204s) [upload=12973]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:09:16.735837421 (12h29m16.325182408s) [upload=12973]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:09:17.021230773 (12h29m16.61057576s) [upload=12973]: Done uploading segments... Aug 7 22:09:17.021247365 (12h29m16.610592352s) [upload=12973]: Upload finished. err: <nil> Aug 7 22:10:03.229004755 (12h30m2.818349752s) [upload=12987]: Starting upload Aug 7 22:10:03.229086592 (12h30m2.818431589s) [upload=12987]: Done starting upload Aug 7 22:10:03.229165603 (12h30m2.8185106s) [upload=12987]: Uploading segments... Aug 7 22:10:03.229268600 (12h30m2.818613618s) [upload=12987]: Got next segment. Inline: true Aug 7 22:10:03.229296584 (12h30m2.818641581s) [upload=12987]: Waiting for error group managing segments... Aug 7 22:10:03.229348544 (12h30m2.818693531s) [upload=12987]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:10:03.229354835 (12h30m2.818699822s) [upload=12987]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:10:03.229357520 (12h30m2.818702517s) [upload=12987]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:03.732013236 (12h30m3.321358233s) [upload=12987]: Done uploading segments... Aug 7 22:10:03.732033424 (12h30m3.321378411s) [upload=12987]: Upload finished. err: <nil> Aug 7 22:11:11.152680180 (12h31m10.742025167s) [upload=13005]: Starting upload Aug 7 22:11:11.152732820 (12h31m10.742077807s) [upload=13005]: Done starting upload Aug 7 22:11:11.152799828 (12h31m10.742144815s) [upload=13005]: Uploading segments... Aug 7 22:11:11.152844534 (12h31m10.742189531s) [upload=13005]: Got next segment. Inline: true Aug 7 22:11:11.152848501 (12h31m10.742193489s) [upload=13005]: Waiting for error group managing segments... Aug 7 22:11:11.152885622 (12h31m10.742230609s) [upload=13005]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:11:11.152889760 (12h31m10.742234747s) [upload=13005]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:11:11.152893338 (12h31m10.742238325s) [upload=13005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:12:10.507162881 (12h32m10.096507878s) [upload=13005]: Done uploading segments... Aug 7 22:12:10.507172128 (12h32m10.096517115s) [upload=13005]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:12:11.394499496 (12h32m10.983844483s) [upload=13024]: Starting upload Aug 7 22:12:11.394565451 (12h32m10.983910429s) [upload=13024]: Done starting upload Aug 7 22:12:11.394614746 (12h32m10.983959733s) [upload=13024]: Uploading segments... Aug 7 22:12:11.394660623 (12h32m10.98400561s) [upload=13024]: Got next segment. Inline: true Aug 7 22:12:11.394666585 (12h32m10.984011572s) [upload=13024]: Waiting for error group managing segments... Aug 7 22:12:11.394716149 (12h32m10.984061136s) [upload=13024]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:11.394722612 (12h32m10.9840676s) [upload=13024]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:12:11.394727622 (12h32m10.984072609s) [upload=13024]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:13:10.756105924 (12h33m10.345450911s) [upload=13024]: Done uploading segments... Aug 7 22:13:10.756126203 (12h33m10.34547118s) [upload=13024]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:13:11.639973323 (12h33m11.22931831s) [upload=13040]: Starting upload Aug 7 22:13:11.640030482 (12h33m11.229375469s) [upload=13040]: Done starting upload Aug 7 22:13:11.640083043 (12h33m11.22942803s) [upload=13040]: Uploading segments... Aug 7 22:13:11.640114102 (12h33m11.229459089s) [upload=13040]: Got next segment. Inline: true Aug 7 22:13:11.640117879 (12h33m11.229462867s) [upload=13040]: Waiting for error group managing segments... Aug 7 22:13:11.640150943 (12h33m11.22949593s) [upload=13040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:13:11.640155241 (12h33m11.229500228s) [upload=13040]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:13:11.640161784 (12h33m11.229506761s) [upload=13040]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:10.987489973 (12h34m10.57683496s) [upload=13040]: Done uploading segments... Aug 7 22:14:10.987508428 (12h34m10.576853415s) [upload=13040]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:14:11.872992660 (12h34m11.462337647s) [upload=13058]: Starting upload Aug 7 22:14:11.873056512 (12h34m11.462401499s) [upload=13058]: Done starting upload Aug 7 22:14:11.873107790 (12h34m11.462452778s) [upload=13058]: Uploading segments... Aug 7 22:14:11.873141936 (12h34m11.462486923s) [upload=13058]: Got next segment. Inline: true Aug 7 22:14:11.873145402 (12h34m11.462490389s) [upload=13058]: Waiting for error group managing segments... Aug 7 22:14:11.873177533 (12h34m11.46252252s) [upload=13058]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:11.873181662 (12h34m11.462526639s) [upload=13058]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:14:11.873183926 (12h34m11.462528913s) [upload=13058]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:15:11.220103017 (12h35m10.809448004s) [upload=13058]: Done uploading segments... Aug 7 22:15:11.220126102 (12h35m10.809471079s) [upload=13058]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:15:12.112939786 (12h35m11.702284773s) [upload=13073]: Starting upload Aug 7 22:15:12.113000451 (12h35m11.702345428s) [upload=13073]: Done starting upload Aug 7 22:15:12.113099632 (12h35m11.702444619s) [upload=13073]: Uploading segments... Aug 7 22:15:12.113176779 (12h35m11.702521766s) [upload=13073]: Got next segment. Inline: true Aug 7 22:15:12.113182540 (12h35m11.702527527s) [upload=13073]: Waiting for error group managing segments... Aug 7 22:15:12.113220422 (12h35m11.702565409s) [upload=13073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:12.113224931 (12h35m11.702569918s) [upload=13073]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:15:12.113227225 (12h35m11.702572212s) [upload=13073]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:16:11.460295293 (12h36m11.04964029s) [upload=13073]: Done uploading segments... Aug 7 22:16:11.460318257 (12h36m11.049663244s) [upload=13073]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:16:12.349841176 (12h36m11.939186153s) [upload=13087]: Starting upload Aug 7 22:16:12.349901702 (12h36m11.939246689s) [upload=13087]: Done starting upload Aug 7 22:16:12.349951537 (12h36m11.939296514s) [upload=13087]: Uploading segments... Aug 7 22:16:12.349987746 (12h36m11.939332733s) [upload=13087]: Got next segment. Inline: true Aug 7 22:16:12.349991694 (12h36m11.939336671s) [upload=13087]: Waiting for error group managing segments... Aug 7 22:16:12.350036460 (12h36m11.939381447s) [upload=13087]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:16:12.350040838 (12h36m11.939385825s) [upload=13087]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:16:12.350043353 (12h36m11.93938833s) [upload=13087]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:17:11.694513348 (12h37m11.283858325s) [upload=13087]: Done uploading segments... Aug 7 22:17:11.694535199 (12h37m11.283880176s) [upload=13087]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.000876344s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 25 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXX...X................................XXXXXXXXX] duration: 30.001323671s FAIL (2025-08-07 22:17:08 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.001341384s core uplink us1
checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway) antiblip: 8 out of 60 is failed (threshold=3) [XX.XX..X.XX...............................................X.] duration: 234.382281ms remote: "tcp:207.211.208.129:443" target: https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222 FAIL (2025-08-07 22:17:05 UTC) orbiter warning: Get "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222": context deadline exceeded Duration: 234.398613ms edge gateway vhost bgp us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXX.X.X.X................................XXXXXX] duration: 30.000592521s FAIL (2025-08-07 22:17:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 2:10:55.341563384 (12h30m54.930908361s) [noise=true , address=77.90.31.10:4520 , piece_id=KSMIVSJ4QSOZAXZN7EVTTC4DGFR2TL4JLJQW2NPYYGWAJDKSZ2ZA, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.391397870 (12h30m54.980742857s) [noise=true , address=45.91.116.14:4189 , piece_id=WTRHRDN2HJN3MQI6334ZHVKLMCT7XH7S42MKYCEK5RY2JHDBESOQ, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.429722625 (12h30m55.019067602s) [noise=true , address=77.90.31.12:4590 , piece_id=QCC2XA4NPGVTUZXRRNXV22QVU77S6QB6R4QEQIYUAFWVON3J35RQ, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.430446209 (12h30m55.019791196s) [noise=true , address=104.36.82.6:4873 , piece_id=FIZQQ56RDZJUYC2JM24K5UMWTQI64SLS56HH54F4EUIXCWH46FOQ, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.438319623 (12h30m55.0276646s) [noise=true , address=45.91.116.5:4298 , piece_id=XJIIUMO7PI6XTBLXE7IRPS2YYI53MRGPEW7HS5CWFXUSIXBT5MBQ, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.440502008 (12h30m55.029846995s) [noise=true , address=45.91.116.13:4154 , piece_id=TPFKH6V3LAM6B2O7L744BZPLT64I2BZVYJXFPEPLIGMVEQ7246YQ, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.449485475 (12h30m55.038830462s) [noise=true , address=104.36.82.7:4909 , piece_id=6ZPKL2IEZN54MOXLAC7P4GURLWMJ7MUBNCBXPUGF6TD6257G7TAA, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.449773497 (12h30m55.039118484s) [noise=true , address=77.90.31.13:4627 , piece_id=3IQP6QL756UL7CNL357KLNUTHQCSEG5337GUT2PD2AS623PTIYCA, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.486432234 (12h30m55.075777221s) [noise=true , address=77.90.31.11:4555 , piece_id=A6WWFITSRKE4GVTQODOYU36ORK7OVE3MVVR7B4FPSSP64DATOQ6A, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.522908903 (12h30m55.11225389s) [noise=true , address=77.90.31.18:4805 , piece_id=VQ3CGKCPEKYPCAIOHQ2FZSINJBHBCCQUZD2AD4AIHX72SZN7SEKA, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.546118308 (12h30m55.135463295s) [noise=true , address=104.36.82.4:4802 , piece_id=B4Q65J747GHE2ZRZDM7RLWC3AQC7TLK5D53REAUA5DMHT3TGA44Q, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.568801126 (12h30m55.158146113s) [noise=true , address=104.36.82.4:4804 , piece_id=BI6ZPR56BKHTA2QGM3BNUBFJM6PAD4K27P5TYHRGLG4NBD56LC4A, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.587208293 (12h30m55.176553341s) [noise=true , address=77.90.31.17:4770 , piece_id=BG2ZL7XC52NEJC5FRS4V75MEPJH34B27IPLTJNQLSX4N3OZGVHXA, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.587245686 (12h30m55.176590713s) [seg_pos={0 0}, upload=13000]: Segment reached optimal threshold of 13 pieces. Aug 7 22:10:55.587371516 (12h30m55.176716533s) [noise=true , address=45.91.116.4:4939 , piece_id=FQ4Z6Y4KIOGJLPCZFVPCUJSYOMGDOTZNAIEKGNWKASCLKOWLCTUQ, seg_pos={0 0}, upload=13000]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Aug 7 22:10:55.587442001 (12h30m55.176786988s) [noise=true , address=45.91.116.7:4009 , piece_id=47X2WBQ4XYRNAZDBVU572QSRHXUNSYRCOUALM4SF4WRHCDW6QTCA, seg_pos={0 0}, upload=13000]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Aug 7 22:10:55.587644678 (12h30m55.176989726s) [noise=true , address=104.36.82.12:4150 , piece_id=XG7E3FHDR2VCPREGTWFFJ52JLGUPRUKQJ75KFYEUXPI4OUNTB3ZA, seg_pos={0 0}, upload=13000]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: context canceled; context canceled Aug 7 22:10:55.587681550 (12h30m55.177026557s) [seg_pos={0 0}, upload=13000]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 22:10:55.587777072 (12h30m55.177122109s) [upload=13000]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:10:55.587787912 (12h30m55.177132929s) [upload=13000]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:55.887211865 (12h30m55.476556883s) [upload=13000]: Done uploading segments... Aug 7 22:10:55.887228857 (12h30m55.476573844s) [upload=13000]: Upload finished. err: <nil> Aug 7 22:11:40.872949328 (12h31m40.462294315s) [upload=13014]: Starting upload Aug 7 22:11:40.873003161 (12h31m40.462348148s) [upload=13014]: Done starting upload Aug 7 22:11:40.873072924 (12h31m40.462417911s) [upload=13014]: Uploading segments... Aug 7 22:11:40.873431210 (12h31m40.462776287s) [upload=13014]: Got next segment. Inline: false Aug 7 22:11:40.873471537 (12h31m40.462816525s) [upload=13014]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:11:40.873480314 (12h31m40.462825331s) [upload=13014]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:12:10.229450117 (12h32m9.818795124s) [upload=13014]: Next segment err: context deadline exceeded Aug 7 22:12:10.229589654 (12h32m9.818934631s) [upload=13014]: Done uploading segments... Aug 7 22:12:10.229603972 (12h32m9.818948949s) [upload=13014]: Upload finished. err: context deadline exceeded Aug 7 22:12:48.993762773 (12h32m48.58310776s) [upload=13033]: Starting upload Aug 7 22:12:48.993844269 (12h32m48.583189246s) [upload=13033]: Done starting upload Aug 7 22:12:48.993904404 (12h32m48.583249391s) [upload=13033]: Uploading segments... Aug 7 22:12:48.994022439 (12h32m48.583367427s) [upload=13033]: Got next segment. Inline: false Aug 7 22:12:48.994048309 (12h32m48.583393296s) [upload=13033]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:48.994052647 (12h32m48.583397634s) [upload=13033]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:13:18.347381136 (12h33m17.936726133s) [upload=13033]: Next segment err: context deadline exceeded Aug 7 22:13:18.347514681 (12h33m17.936859668s) [upload=13033]: Done uploading segments... Aug 7 22:13:18.347529670 (12h33m17.936874657s) [upload=13033]: Upload finished. err: context deadline exceeded Aug 7 22:13:51.696413792 (12h33m51.285758769s) [upload=13052]: Starting upload Aug 7 22:13:51.696517580 (12h33m51.285862567s) [upload=13052]: Done starting upload Aug 7 22:13:51.696601842 (12h33m51.285946829s) [upload=13052]: Uploading segments... Aug 7 22:13:51.696787407 (12h33m51.286132414s) [upload=13052]: Got next segment. Inline: false Aug 7 22:13:51.696823415 (12h33m51.286168413s) [upload=13052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:13:51.696829066 (12h33m51.286174073s) [upload=13052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:14:21.042712175 (12h34m20.632057152s) [upload=13052]: Next segment err: context deadline exceeded Aug 7 22:14:21.042793340 (12h34m20.632138327s) [upload=13052]: Done uploading segments... Aug 7 22:14:21.042809000 (12h34m20.632153987s) [upload=13052]: Upload finished. err: context deadline exceeded Aug 7 22:14:36.252577987 (12h34m35.841922974s) [upload=13065]: Starting upload Aug 7 22:14:36.252631760 (12h34m35.841976747s) [upload=13065]: Done starting upload Aug 7 22:14:36.252692656 (12h34m35.842037643s) [upload=13065]: Uploading segments... Aug 7 22:14:36.252805532 (12h34m35.842150519s) [upload=13065]: Got next segment. Inline: false Aug 7 22:14:36.252856870 (12h34m35.842201857s) [upload=13065]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:36.252863202 (12h34m35.842208189s) [upload=13065]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:15:05.607209972 (12h35m5.196554959s) [upload=13065]: Next segment err: context deadline exceeded Aug 7 22:15:05.607234078 (12h35m5.196579065s) [upload=13065]: Done uploading segments... Aug 7 22:15:05.607262002 (12h35m5.196606989s) [upload=13065]: Upload finished. err: context deadline exceeded Aug 7 22:15:30.794674976 (12h35m30.384019963s) [upload=13077]: Starting upload Aug 7 22:15:30.794787000 (12h35m30.384132008s) [upload=13077]: Done starting upload Aug 7 22:15:30.794881030 (12h35m30.384226007s) [upload=13077]: Uploading segments... Aug 7 22:15:30.795138724 (12h35m30.384483711s) [upload=13077]: Got next segment. Inline: false Aug 7 22:15:30.795218847 (12h35m30.384563834s) [upload=13077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:30.795225099 (12h35m30.384570106s) [upload=13077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:16:00.147601500 (12h35m59.736946487s) [upload=13077]: Next segment err: context deadline exceeded Aug 7 22:16:00.147674640 (12h35m59.737019628s) [upload=13077]: Done uploading segments... Aug 7 22:16:00.147688897 (12h35m59.737033884s) [upload=13077]: Upload finished. err: context deadline exceeded Aug 7 22:16:34.181460693 (12h36m33.77080568s) [upload=13093]: Starting upload Aug 7 22:16:34.181519996 (12h36m33.770864983s) [upload=13093]: Done starting upload Aug 7 22:16:34.181628824 (12h36m33.770973811s) [upload=13093]: Uploading segments... Aug 7 22:16:34.181727263 (12h36m33.77107226s) [upload=13093]: Got next segment. Inline: false Aug 7 22:16:34.181766387 (12h36m33.771111385s) [upload=13093]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:16:34.181772269 (12h36m33.771117267s) [upload=13093]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:17:03.528495677 (12h37m3.117840664s) [upload=13093]: Next segment err: context deadline exceeded Aug 7 22:17:03.528680499 (12h37m3.118025497s) [upload=13093]: Done uploading segments... Aug 7 22:17:03.528713574 (12h37m3.118058551s) [upload=13093]: Upload finished. err: context deadline exceeded Duration: 30.000613712s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.........X.................X................................] duration: 1.866692139s FAIL (2025-08-07 22:17:02 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== 8m27.919686751s) [upload=12612]: Done uploading segments... Aug 7 21:48:28.330358527 (12h8m27.919703514s) [upload=12612]: Upload finished. err: <nil> Aug 7 21:48:54.943937458 (12h8m54.533282445s) [upload=12618]: Starting upload Aug 7 21:48:54.944002622 (12h8m54.533347609s) [upload=12618]: Done starting upload Aug 7 21:48:54.944106100 (12h8m54.533451087s) [upload=12618]: Uploading segments... Aug 7 21:48:54.944196242 (12h8m54.53354132s) [upload=12618]: Got next segment. Inline: true Aug 7 21:48:54.944206543 (12h8m54.53355158s) [upload=12618]: Waiting for error group managing segments... Aug 7 21:48:54.944254484 (12h8m54.533599461s) [upload=12618]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:48:54.944259223 (12h8m54.5336042s) [upload=12618]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:48:54.944262139 (12h8m54.533607126s) [upload=12618]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:48:55.330753187 (12h8m54.920098174s) [upload=12618]: Done uploading segments... Aug 7 21:48:55.330774959 (12h8m54.920119936s) [upload=12618]: Upload finished. err: <nil> Aug 7 21:49:36.650951942 (12h9m36.240296929s) [upload=12631]: Starting upload Aug 7 21:49:36.650993252 (12h9m36.240338239s) [upload=12631]: Done starting upload Aug 7 21:49:36.651049029 (12h9m36.240394016s) [upload=12631]: Uploading segments... Aug 7 21:49:36.651075309 (12h9m36.240420286s) [upload=12631]: Got next segment. Inline: true Aug 7 21:49:36.651078775 (12h9m36.240423763s) [upload=12631]: Waiting for error group managing segments... Aug 7 21:49:36.651115776 (12h9m36.240460773s) [upload=12631]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:49:36.651120254 (12h9m36.240465251s) [upload=12631]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:49:36.651124473 (12h9m36.24046947s) [upload=12631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:49:36.920683881 (12h9m36.510028868s) [upload=12631]: Done uploading segments... Aug 7 21:49:36.920707246 (12h9m36.510052233s) [upload=12631]: Upload finished. err: <nil> Aug 7 21:50:03.363429937 (12h10m2.952774924s) [upload=12636]: Starting upload Aug 7 21:50:03.363493408 (12h10m2.952838405s) [upload=12636]: Done starting upload Aug 7 21:50:03.363582518 (12h10m2.952927515s) [upload=12636]: Uploading segments... Aug 7 21:50:03.363692078 (12h10m2.953037085s) [upload=12636]: Got next segment. Inline: true Aug 7 21:50:03.363699752 (12h10m2.953044739s) [upload=12636]: Waiting for error group managing segments... Aug 7 21:50:03.363764116 (12h10m2.953109103s) [upload=12636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:50:03.363773323 (12h10m2.95311831s) [upload=12636]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:50:03.363777271 (12h10m2.953122258s) [upload=12636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:50:03.636022787 (12h10m3.225367774s) [upload=12636]: Done uploading segments... Aug 7 21:50:03.636042204 (12h10m3.225387202s) [upload=12636]: Upload finished. err: <nil> Aug 7 21:50:20.944550175 (12h10m20.533895152s) [upload=12642]: Starting upload Aug 7 21:50:20.944594911 (12h10m20.533939898s) [upload=12642]: Done starting upload Aug 7 21:50:20.944644405 (12h10m20.533989392s) [upload=12642]: Uploading segments... Aug 7 21:50:20.944674473 (12h10m20.534019461s) [upload=12642]: Got next segment. Inline: true Aug 7 21:50:20.944677820 (12h10m20.534022807s) [upload=12642]: Waiting for error group managing segments... Aug 7 21:50:20.944709450 (12h10m20.534054437s) [upload=12642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:50:20.944715131 (12h10m20.534060118s) [upload=12642]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:50:20.944717505 (12h10m20.534062493s) [upload=12642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:50:21.212077476 (12h10m20.801422473s) [upload=12642]: Done uploading segments... Aug 7 21:50:21.212101602 (12h10m20.801446599s) [upload=12642]: Upload finished. err: <nil> Aug 7 21:51:22.649621444 (12h11m22.238966431s) [upload=12660]: Starting upload Aug 7 21:51:22.649696837 (12h11m22.239041824s) [upload=12660]: Done starting upload Aug 7 21:51:22.649777742 (12h11m22.239122729s) [upload=12660]: Uploading segments... Aug 7 21:51:22.649867044 (12h11m22.239212041s) [upload=12660]: Got next segment. Inline: true Aug 7 21:51:22.649872053 (12h11m22.23921704s) [upload=12660]: Waiting for error group managing segments... Aug 7 21:51:22.649940053 (12h11m22.23928504s) [upload=12660]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:51:22.649946785 (12h11m22.239291772s) [upload=12660]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:51:22.649949932 (12h11m22.239294919s) [upload=12660]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:51:22.914527578 (12h11m22.503872565s) [upload=12660]: Done uploading segments... Aug 7 21:51:22.914549130 (12h11m22.503894117s) [upload=12660]: Upload finished. err: <nil> Aug 7 21:51:38.628060070 (12h11m38.217405057s) [upload=12665]: Starting upload Aug 7 21:51:38.628100548 (12h11m38.217445535s) [upload=12665]: Done starting upload Aug 7 21:51:38.628155373 (12h11m38.21750036s) [upload=12665]: Uploading segments... Aug 7 21:51:38.628187814 (12h11m38.217532802s) [upload=12665]: Got next segment. Inline: true Aug 7 21:51:38.628191381 (12h11m38.217536368s) [upload=12665]: Waiting for error group managing segments... Aug 7 21:51:38.628222691 (12h11m38.217567678s) [upload=12665]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:51:38.628226268 (12h11m38.217571255s) [upload=12665]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:51:38.628228662 (12h11m38.217573649s) [upload=12665]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:51:38.940125222 (12h11m38.529470269s) [upload=12665]: Done uploading segments... Aug 7 21:51:38.940147124 (12h11m38.529492191s) [upload=12665]: Upload finished. err: <nil> Aug 7 21:52:16.262879794 (12h12m15.852224781s) [upload=12676]: Starting upload Aug 7 21:52:16.262939168 (12h12m15.852284155s) [upload=12676]: Done starting upload Aug 7 21:52:16.263047876 (12h12m15.852392873s) [upload=12676]: Uploading segments... Aug 7 21:52:16.263139601 (12h12m15.852484598s) [upload=12676]: Got next segment. Inline: true Aug 7 21:52:16.263144440 (12h12m15.852489417s) [upload=12676]: Waiting for error group managing segments... Aug 7 21:52:16.263191931 (12h12m15.852536918s) [upload=12676]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:52:16.263197472 (12h12m15.852542449s) [upload=12676]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:52:16.263200056 (12h12m15.852545043s) [upload=12676]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:52:16.570506954 (12h12m16.159851931s) [upload=12676]: Done uploading segments... Aug 7 21:52:16.570524978 (12h12m16.159869965s) [upload=12676]: Upload finished. err: <nil> Aug 7 21:53:00.574165558 (12h13m0.163510545s) [upload=12691]: Starting upload Aug 7 21:53:00.574212286 (12h13m0.163557273s) [upload=12691]: Done starting upload Aug 7 21:53:00.574248476 (12h13m0.163593463s) [upload=12691]: Uploading segments... Aug 7 21:53:00.574327307 (12h13m0.163672294s) [upload=12691]: Got next segment. Inline: true Aug 7 21:53:00.574332868 (12h13m0.163677855s) [upload=12691]: Waiting for error group managing segments... Aug 7 21:53:00.574373505 (12h13m0.163718492s) [upload=12691]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:53:00.574390598 (12h13m0.163735585s) [upload=12691]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:53:00.574393373 (12h13m0.16373835s) [upload=12691]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:53:00.840335180 (12h13m0.429680167s) [upload=12691]: Done uploading segments... Aug 7 21:53:00.840351500 (12h13m0.429696487s) [upload=12691]: Upload finished. err: <nil> Aug 7 21:53:46.136340579 (12h13m45.725685566s) [upload=12705]: Starting upload Aug 7 21:53:46.136382029 (12h13m45.725727016s) [upload=12705]: Done starting upload Aug 7 21:53:46.136495205 (12h13m45.725840192s) [upload=12705]: Uploading segments... Aug 7 21:53:46.136529601 (12h13m45.725874588s) [upload=12705]: Got next segment. Inline: true Aug 7 21:53:46.136533228 (12h13m45.725878215s) [upload=12705]: Waiting for error group managing segments... Aug 7 21:53:46.136581701 (12h13m45.725926688s) [upload=12705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:53:46.136585949 (12h13m45.725930936s) [upload=12705]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:53:46.136588393 (12h13m45.72593338s) [upload=12705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:53:46.427021148 (12h13m46.016366155s) [upload=12705]: Done uploading segments... Aug 7 21:53:46.427045194 (12h13m46.016390181s) [upload=12705]: Upload finished. err: <nil> Aug 7 21:54:41.670591536 (12h14m41.259936523s) [upload=12721]: Starting upload Aug 7 21:54:41.670632916 (12h14m41.259977903s) [upload=12721]: Done starting upload Aug 7 21:54:41.670677612 (12h14m41.260022599s) [upload=12721]: Uploading segments... Aug 7 21:54:41.670708571 (12h14m41.260053558s) [upload=12721]: Got next segment. Inline: true Aug 7 21:54:41.670713510 (12h14m41.260058497s) [upload=12721]: Waiting for error group managing segments... Aug 7 21:54:41.670746032 (12h14m41.260091019s) [upload=12721]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:54:41.670749979 (12h14m41.260094966s) [upload=12721]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:54:41.670752424 (12h14m41.260097411s) [upload=12721]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:54:41.940759391 (12h14m41.530104368s) [upload=12721]: Done uploading segments... Aug 7 21:54:41.940776734 (12h14m41.530121721s) [upload=12721]: Upload finished. err: <nil> Duration: 1.866718128s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [XXXXXXXXXXXXX.......X................................XXXXXXX] duration: 30.000435731s FAIL (2025-08-07 22:16:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000526774s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 19 out of 60 is failed (threshold=3) [XXXXXXXXXXX.X........................................XXXXXXX] duration: 30.001112286s FAIL (2025-08-07 22:16:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001132254s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 25 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXX...X................................XXXXXXXX] duration: 30.000596098s FAIL (2025-08-07 22:16:37 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000608991s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 21 out of 60 is failed (threshold=3) [XXXXXXXXXXXXX.X...X...................................XXXXXX] duration: 30.001333575s FAIL (2025-08-07 22:16:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 2]: Starting upload Aug 7 22:09:14.345291719 (12h29m13.934636706s) [upload=12972]: Done starting upload Aug 7 22:09:14.345358137 (12h29m13.934703114s) [upload=12972]: Uploading segments... Aug 7 22:09:14.345388645 (12h29m13.934733632s) [upload=12972]: Got next segment. Inline: true Aug 7 22:09:14.345392141 (12h29m13.934737118s) [upload=12972]: Waiting for error group managing segments... Aug 7 22:09:14.345426327 (12h29m13.934771314s) [upload=12972]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:09:14.345430324 (12h29m13.934775301s) [upload=12972]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:09:14.345432839 (12h29m13.934777826s) [upload=12972]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:09:14.834006850 (12h29m14.423351857s) [upload=12972]: Done uploading segments... Aug 7 22:09:14.834026347 (12h29m14.423371334s) [upload=12972]: Upload finished. err: <nil> Aug 7 22:09:53.131289328 (12h29m52.720634315s) [upload=12982]: Starting upload Aug 7 22:09:53.131364442 (12h29m52.720709489s) [upload=12982]: Done starting upload Aug 7 22:09:53.131436210 (12h29m52.720781197s) [upload=12982]: Uploading segments... Aug 7 22:09:53.131465205 (12h29m52.720810192s) [upload=12982]: Got next segment. Inline: true Aug 7 22:09:53.131468902 (12h29m52.720813889s) [upload=12982]: Waiting for error group managing segments... Aug 7 22:09:53.131498809 (12h29m52.720843796s) [upload=12982]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:09:53.131503017 (12h29m52.720848004s) [upload=12982]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:09:53.131505141 (12h29m52.720850118s) [upload=12982]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:09:53.419040148 (12h29m53.008385125s) [upload=12982]: Done uploading segments... Aug 7 22:09:53.419058042 (12h29m53.008403029s) [upload=12982]: Upload finished. err: <nil> Aug 7 22:10:12.329552115 (12h30m11.918897102s) [upload=12989]: Starting upload Aug 7 22:10:12.330067942 (12h30m11.919412929s) [upload=12989]: Done starting upload Aug 7 22:10:12.330139077 (12h30m11.919484054s) [upload=12989]: Uploading segments... Aug 7 22:10:12.330186228 (12h30m11.919531255s) [upload=12989]: Got next segment. Inline: true Aug 7 22:10:12.330192710 (12h30m11.919537737s) [upload=12989]: Waiting for error group managing segments... Aug 7 22:10:12.330240762 (12h30m11.919585749s) [upload=12989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:10:12.330248126 (12h30m11.919593113s) [upload=12989]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:10:12.330252083 (12h30m11.91959709s) [upload=12989]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:12.591107606 (12h30m12.180452593s) [upload=12989]: Done uploading segments... Aug 7 22:10:12.591124920 (12h30m12.180469907s) [upload=12989]: Upload finished. err: <nil> Aug 7 22:10:57.486329607 (12h30m57.075674594s) [upload=13003]: Starting upload Aug 7 22:10:57.486414950 (12h30m57.075759927s) [upload=13003]: Done starting upload Aug 7 22:10:57.486491847 (12h30m57.075836834s) [upload=13003]: Uploading segments... Aug 7 22:10:57.486573694 (12h30m57.075918681s) [upload=13003]: Got next segment. Inline: true Aug 7 22:10:57.486578242 (12h30m57.07592322s) [upload=13003]: Waiting for error group managing segments... Aug 7 22:10:57.486645681 (12h30m57.075990658s) [upload=13003]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:10:57.486650361 (12h30m57.075995348s) [upload=13003]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:10:57.486652795 (12h30m57.075997782s) [upload=13003]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:57.751495816 (12h30m57.340840803s) [upload=13003]: Done uploading segments... Aug 7 22:10:57.751515013 (12h30m57.34086s) [upload=13003]: Upload finished. err: <nil> Aug 7 22:11:44.715949867 (12h31m44.305294854s) [upload=13015]: Starting upload Aug 7 22:11:44.716016213 (12h31m44.3053612s) [upload=13015]: Done starting upload Aug 7 22:11:44.716066600 (12h31m44.305411587s) [upload=13015]: Uploading segments... Aug 7 22:11:44.716104322 (12h31m44.305449309s) [upload=13015]: Got next segment. Inline: true Aug 7 22:11:44.716107879 (12h31m44.305452866s) [upload=13015]: Waiting for error group managing segments... Aug 7 22:11:44.716139289 (12h31m44.305484276s) [upload=13015]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:11:44.716142826 (12h31m44.305487803s) [upload=13015]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:11:44.716146493 (12h31m44.30549147s) [upload=13015]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:12:14.067179909 (12h32m13.656524896s) [upload=13015]: Done uploading segments... Aug 7 22:12:14.067200478 (12h32m13.656545465s) [upload=13015]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:12:53.398168829 (12h32m52.987513816s) [upload=13035]: Starting upload Aug 7 22:12:53.398221350 (12h32m52.987566337s) [upload=13035]: Done starting upload Aug 7 22:12:53.398263170 (12h32m52.987608147s) [upload=13035]: Uploading segments... Aug 7 22:12:53.398291284 (12h32m52.987636261s) [upload=13035]: Got next segment. Inline: true Aug 7 22:12:53.398295111 (12h32m52.987640088s) [upload=13035]: Waiting for error group managing segments... Aug 7 22:12:53.398338094 (12h32m52.987683081s) [upload=13035]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:53.398344366 (12h32m52.987689363s) [upload=13035]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:12:53.398348754 (12h32m52.987693771s) [upload=13035]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:13:22.757670742 (12h33m22.347015729s) [upload=13035]: Done uploading segments... Aug 7 22:13:22.757697282 (12h33m22.34704227s) [upload=13035]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:14:04.336472471 (12h34m3.925817458s) [upload=13057]: Starting upload Aug 7 22:14:04.336528488 (12h34m3.925873475s) [upload=13057]: Done starting upload Aug 7 22:14:04.336571460 (12h34m3.925916447s) [upload=13057]: Uploading segments... Aug 7 22:14:04.336601788 (12h34m3.925946775s) [upload=13057]: Got next segment. Inline: true Aug 7 22:14:04.336605075 (12h34m3.925950052s) [upload=13057]: Waiting for error group managing segments... Aug 7 22:14:04.336639170 (12h34m3.925984147s) [upload=13057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:04.336643388 (12h34m3.925988375s) [upload=13057]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:14:04.336645732 (12h34m3.925990719s) [upload=13057]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:33.693506183 (12h34m33.28285116s) [upload=13057]: Done uploading segments... Aug 7 22:14:33.693533705 (12h34m33.282878692s) [upload=13057]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:14:56.898136222 (12h34m56.487481239s) [upload=13071]: Starting upload Aug 7 22:14:56.898259367 (12h34m56.487604384s) [upload=13071]: Done starting upload Aug 7 22:14:56.898331565 (12h34m56.487676583s) [upload=13071]: Uploading segments... Aug 7 22:14:56.898390859 (12h34m56.487735866s) [upload=13071]: Got next segment. Inline: true Aug 7 22:14:56.898397631 (12h34m56.487742649s) [upload=13071]: Waiting for error group managing segments... Aug 7 22:14:56.898453538 (12h34m56.487798535s) [upload=13071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:56.898460371 (12h34m56.487805388s) [upload=13071]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:14:56.898467315 (12h34m56.487812332s) [upload=13071]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:15:26.250806715 (12h35m25.840151692s) [upload=13071]: Done uploading segments... Aug 7 22:15:26.250836313 (12h35m25.8401813s) [upload=13071]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:15:34.770168401 (12h35m34.359513388s) [upload=13080]: Starting upload Aug 7 22:15:34.770290034 (12h35m34.359635021s) [upload=13080]: Done starting upload Aug 7 22:15:34.770415243 (12h35m34.35976023s) [upload=13080]: Uploading segments... Aug 7 22:15:34.770513081 (12h35m34.359858068s) [upload=13080]: Got next segment. Inline: true Aug 7 22:15:34.770518170 (12h35m34.359863157s) [upload=13080]: Waiting for error group managing segments... Aug 7 22:15:34.770576111 (12h35m34.359921098s) [upload=13080]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:34.770582192 (12h35m34.359927169s) [upload=13080]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:15:34.770584747 (12h35m34.359929734s) [upload=13080]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:16:04.123598264 (12h36m3.712943251s) [upload=13080]: Done uploading segments... Aug 7 22:16:04.123618482 (12h36m3.712963469s) [upload=13080]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:16:05.006396275 (12h36m4.595741272s) [upload=13084]: Starting upload Aug 7 22:16:05.006480486 (12h36m4.595825473s) [upload=13084]: Done starting upload Aug 7 22:16:05.006520022 (12h36m4.595865009s) [upload=13084]: Uploading segments... Aug 7 22:16:05.006575979 (12h36m4.595920966s) [upload=13084]: Got next segment. Inline: true Aug 7 22:16:05.006582241 (12h36m4.595927228s) [upload=13084]: Waiting for error group managing segments... Aug 7 22:16:05.006619191 (12h36m4.595964178s) [upload=13084]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:16:05.006623339 (12h36m4.595968326s) [upload=13084]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:16:05.006625714 (12h36m4.595970701s) [upload=13084]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:16:34.356940910 (12h36m33.946285897s) [upload=13084]: Done uploading segments... Aug 7 22:16:34.356970046 (12h36m33.946315033s) [upload=13084]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.001342171s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [......X..X....................X....................X........] duration: 6.72593351s FAIL (2025-08-07 22:16:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 34e70bda2fe1d7206098dbfb59fc4576ab613f2b752f601da4a5b93473be374e != before ef9ba24509b0003982578e9daec1e6051f8d6ac0bef4f9769b005a8c713045cf ======LOGS====== ding segments... Aug 7 22:10:42.792858699 (12h30m42.382203686s) [upload=12997]: Got next segment. Inline: false Aug 7 22:10:42.792883667 (12h30m42.382228644s) [upload=12997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:10:42.792887915 (12h30m42.382232902s) [upload=12997]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:10:43.239239407 (12h30m42.828584384s) [seg_pos={0 0}, upload=12997]: Begin upload segment... Aug 7 22:10:43.239300844 (12h30m42.828645831s) [seg_pos={0 0}, upload=12997]: Done begin upload segment. Aug 7 22:10:43.239329008 (12h30m42.828673995s) [noise=true , address=104.36.82.61:4394 , piece_id=BZ4OH34HU6COIEBD542MEZ5YGPXPLFMQKIO2EOV23UBTDTFK5XAA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.239541695 (12h30m42.828886692s) [noise=true , address=104.36.82.37:4687 , piece_id=I7ZESSKAS4ZUYX3CUI4ZRZWY2PLRATRNR6FGVGVAXQD6JT3DRUIA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.239923255 (12h30m42.829268242s) [noise=true , address=104.36.82.10:4087 , piece_id=PDZWS5ZZVKC5BCXJIY43NBQVXM7NCLKJZNSVTD3G2K6F2YEY4RVA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.240056509 (12h30m42.829401506s) [noise=true , address=45.91.116.6:4995 , piece_id=GYQIOSTEVRSY6VKX3GQCUC5YZY5S3ZV7R6X2LBG3QRTOOQD53NOA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.240324512 (12h30m42.829669499s) [noise=true , address=104.36.82.38:4705 , piece_id=M557M36RJRMMND4F33G7BOBYVMESQZZN7ZFESJ5CLHEKXWBCL5SA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.240397562 (12h30m42.829742549s) [noise=true , address=79.127.201.215:4351 , piece_id=367MVHSAIIGC6UZQY4EMH25DJCYYTAIUMPFF3PY426RZJHIO44DA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.240579942 (12h30m42.829924929s) [noise=true , address=104.36.82.47:4524 , piece_id=JDRSOKIHNV2RD3COKNJQHRYG2OXOPUBQOG3AHCTYOKKYC3LZISNQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.240814029 (12h30m42.830159016s) [noise=true , address=64.49.13.169:4710 , piece_id=KOUJTCIUNCXFZSJWG6ZKR4KV42WV3ABECSARKOQ5YB5W4LAJTGAA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.240831132 (12h30m42.830176119s) [noise=true , address=104.36.82.59:4331 , piece_id=54OHTXRTR2J4S4HECSB7DFPHSC7T4UGBBC4X7WI72IVCMESI5BQQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.241273939 (12h30m42.830618926s) [noise=true , address=104.36.82.44:4881 , piece_id=YWXSTJSJ46ERQLZ6A2E36CX3R2FI4XVVEZGUMDYJX4ZG3M3R5ACQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.241589863 (12h30m42.83093485s) [noise=true , address=104.36.82.2:4750 , piece_id=24FHN36X2VIWLTYUDHSJNLSOHHEZ56K6DBIGSMI2ZCA26CPT7L7Q, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.241938992 (12h30m42.831283979s) [noise=true , address=104.36.82.3:4791 , piece_id=DJ46BVH37DMTY3YD6VG2AEPNNTPYF7EH4IA3Y633UH3QO6XUTKPQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.242014867 (12h30m42.831359854s) [noise=true , address=104.36.82.58:4270 , piece_id=HLYO6DZ75CNGFXJ6DA5SKQNVPD5COMDHFAETYUZTBOSKMPT4BUEQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.242250207 (12h30m42.831595194s) [noise=true , address=104.36.82.44:4885 , piece_id=FJ726HA3PDRWAYDFGUEK2Y7D3B77ERCFPJSGWNKKPLGWXC3WK7GA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.242484636 (12h30m42.831829623s) [noise=true , address=79.127.205.246:4254 , piece_id=V3AT6KY72VDSIMMOZUC4JECVAQSPAHMAPC5CXDVCO46Z6RTPRUVA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.242691451 (12h30m42.832036438s) [noise=true , address=104.36.82.8:4995 , piece_id=BI77BF2ZGP3ZMB2ABMBUCKQCEF3Z6QHRC7IT6Q2ATIF6VON32WWQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.242894410 (12h30m42.832239397s) [noise=true , address=104.36.82.43:4859 , piece_id=P2S2T5IIUTG4D6EQILFQ7OR5E5JOZHAYO563X7ED5UBN7THKW4QA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.243127415 (12h30m42.832472402s) [noise=true , address=104.36.82.46:4509 , piece_id=WQ3UWQPCMAZ7EDHTSURUQAKL6KWHUA3GSJI6U2G5ZUABRGCTFEXQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:44.351806366 (12h30m43.941151353s) [upload=12997]: Next returned nil segment Aug 7 22:10:44.351819280 (12h30m43.941164257s) [upload=12997]: Waiting for error group managing segments... Aug 7 22:10:44.651888156 (12h30m44.241233144s) [noise=true , address=79.127.205.246:4254 , piece_id=V3AT6KY72VDSIMMOZUC4JECVAQSPAHMAPC5CXDVCO46Z6RTPRUVA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:44.674573650 (12h30m44.263918637s) [noise=true , address=64.49.13.169:4710 , piece_id=KOUJTCIUNCXFZSJWG6ZKR4KV42WV3ABECSARKOQ5YB5W4LAJTGAA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.186610734 (12h30m44.775955721s) [noise=true , address=104.36.82.38:4705 , piece_id=M557M36RJRMMND4F33G7BOBYVMESQZZN7ZFESJ5CLHEKXWBCL5SA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.204653265 (12h30m44.793998252s) [noise=true , address=104.36.82.58:4270 , piece_id=HLYO6DZ75CNGFXJ6DA5SKQNVPD5COMDHFAETYUZTBOSKMPT4BUEQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.218003397 (12h30m44.807348374s) [noise=true , address=79.127.201.215:4351 , piece_id=367MVHSAIIGC6UZQY4EMH25DJCYYTAIUMPFF3PY426RZJHIO44DA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.276599656 (12h30m44.865944643s) [noise=true , address=104.36.82.46:4509 , piece_id=WQ3UWQPCMAZ7EDHTSURUQAKL6KWHUA3GSJI6U2G5ZUABRGCTFEXQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.316254365 (12h30m44.905599352s) [noise=true , address=104.36.82.2:4750 , piece_id=24FHN36X2VIWLTYUDHSJNLSOHHEZ56K6DBIGSMI2ZCA26CPT7L7Q, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.349441480 (12h30m44.938786467s) [noise=true , address=104.36.82.3:4791 , piece_id=DJ46BVH37DMTY3YD6VG2AEPNNTPYF7EH4IA3Y633UH3QO6XUTKPQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.352466728 (12h30m44.941811715s) [noise=true , address=104.36.82.44:4881 , piece_id=YWXSTJSJ46ERQLZ6A2E36CX3R2FI4XVVEZGUMDYJX4ZG3M3R5ACQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.418760245 (12h30m45.008105232s) [noise=true , address=104.36.82.37:4687 , piece_id=I7ZESSKAS4ZUYX3CUI4ZRZWY2PLRATRNR6FGVGVAXQD6JT3DRUIA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.540632266 (12h30m45.129977253s) [noise=true , address=104.36.82.10:4087 , piece_id=PDZWS5ZZVKC5BCXJIY43NBQVXM7NCLKJZNSVTD3G2K6F2YEY4RVA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.567016576 (12h30m45.156361563s) [noise=true , address=104.36.82.59:4331 , piece_id=54OHTXRTR2J4S4HECSB7DFPHSC7T4UGBBC4X7WI72IVCMESI5BQQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.597615539 (12h30m45.186960526s) [noise=true , address=104.36.82.44:4885 , piece_id=FJ726HA3PDRWAYDFGUEK2Y7D3B77ERCFPJSGWNKKPLGWXC3WK7GA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.676238976 (12h30m45.265584013s) [noise=true , address=45.91.116.6:4995 , piece_id=GYQIOSTEVRSY6VKX3GQCUC5YZY5S3ZV7R6X2LBG3QRTOOQD53NOA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.676277709 (12h30m45.265622696s) [seg_pos={0 0}, upload=12997]: Segment reached optimal threshold of 14 pieces. Aug 7 22:10:45.676633781 (12h30m45.265978758s) [noise=true , address=104.36.82.8:4995 , piece_id=BI77BF2ZGP3ZMB2ABMBUCKQCEF3Z6QHRC7IT6Q2ATIF6VON32WWQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tvoTD1LoSj3q2rLiDckfT56dXNKKwdVu1ZAsBZExNFVTom9tH): protocol: expected piece hash; context canceled; EOF Aug 7 22:10:45.676841539 (12h30m45.266186526s) [noise=true , address=104.36.82.43:4859 , piece_id=P2S2T5IIUTG4D6EQILFQ7OR5E5JOZHAYO563X7ED5UBN7THKW4QA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126SqCtRGPziEeopRjA4nNkjkCZWvUwkt5XaMgcqZKJwoxuYWY7): protocol: rpc: dial tcp 104.36.82.43:4859: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:10:45.676869993 (12h30m45.26621498s) [noise=true , address=104.36.82.47:4524 , piece_id=JDRSOKIHNV2RD3COKNJQHRYG2OXOPUBQOG3AHCTYOKKYC3LZISNQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4): protocol: read tcp 188.166.225.170:58322->104.36.82.47:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:10:45.676908757 (12h30m45.266253744s) [noise=true , address=104.36.82.61:4394 , piece_id=BZ4OH34HU6COIEBD542MEZ5YGPXPLFMQKIO2EOV23UBTDTFK5XAA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dSpTqCxeAcL9PVYbP7BhmuKE7papFPc9swB1Fy83WcFDgQbcF): protocol: rpc: dial tcp 104.36.82.61:4394: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:10:45.676927984 (12h30m45.266272961s) [seg_pos={0 0}, upload=12997]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:10:45.677012455 (12h30m45.266357443s) [upload=12997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:10:45.677016543 (12h30m45.26636152s) [upload=12997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:45.998268658 (12h30m45.587613635s) [upload=12997]: Done uploading segments... Aug 7 22:10:45.998285902 (12h30m45.587630889s) [upload=12997]: Upload finished. err: <nil> Duration: 6.725963406s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 750.104066ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 22:16:15 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 750.150025ms edge gateway vhost us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 27 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXX...X..X............................XXXXX] duration: 1m0.000598503s FAIL (2025-08-07 22:16:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000617218s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 19 out of 60 is failed (threshold=3) [XXXXXXXXXXXX.X........................................XXXXXX] duration: 30.00042815s FAIL (2025-08-07 22:16:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000449822s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 26 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXX.XX..X...............................XXXXX] duration: 1m0.001012604s FAIL (2025-08-07 22:16:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== h27m42.267393349s) [upload=12938]: Starting upload Aug 7 22:07:42.678122394 (12h27m42.267467381s) [upload=12938]: Done starting upload Aug 7 22:07:42.678184523 (12h27m42.26752951s) [upload=12938]: Uploading segments... Aug 7 22:07:42.678220613 (12h27m42.2675656s) [upload=12938]: Got next segment. Inline: true Aug 7 22:07:42.678225311 (12h27m42.267570299s) [upload=12938]: Waiting for error group managing segments... Aug 7 22:07:42.678261721 (12h27m42.267606708s) [upload=12938]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:07:42.678265758 (12h27m42.267610735s) [upload=12938]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:07:42.678268203 (12h27m42.26761319s) [upload=12938]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:07:42.941204890 (12h27m42.530549877s) [upload=12938]: Done uploading segments... Aug 7 22:07:42.941220821 (12h27m42.530565808s) [upload=12938]: Upload finished. err: <nil> Aug 7 22:08:12.564234560 (12h28m12.153579557s) [upload=12951]: Starting upload Aug 7 22:08:12.564392563 (12h28m12.15373757s) [upload=12951]: Done starting upload Aug 7 22:08:12.564478317 (12h28m12.153823314s) [upload=12951]: Uploading segments... Aug 7 22:08:12.564532671 (12h28m12.153877658s) [upload=12951]: Got next segment. Inline: true Aug 7 22:08:12.564537029 (12h28m12.153882006s) [upload=12951]: Waiting for error group managing segments... Aug 7 22:08:12.564572748 (12h28m12.153917735s) [upload=12951]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:08:12.564577227 (12h28m12.153922204s) [upload=12951]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:08:12.564579511 (12h28m12.153924498s) [upload=12951]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:08:12.853863397 (12h28m12.443208384s) [upload=12951]: Done uploading segments... Aug 7 22:08:12.853882363 (12h28m12.44322735s) [upload=12951]: Upload finished. err: <nil> Aug 7 22:08:48.226332068 (12h28m47.815677055s) [upload=12963]: Starting upload Aug 7 22:08:48.226393765 (12h28m47.815738743s) [upload=12963]: Done starting upload Aug 7 22:08:48.226441998 (12h28m47.815786975s) [upload=12963]: Uploading segments... Aug 7 22:08:48.226471044 (12h28m47.815816031s) [upload=12963]: Got next segment. Inline: true Aug 7 22:08:48.226474330 (12h28m47.815819317s) [upload=12963]: Waiting for error group managing segments... Aug 7 22:08:48.226505239 (12h28m47.815850216s) [upload=12963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:08:48.226508785 (12h28m47.815853772s) [upload=12963]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:08:48.226511250 (12h28m47.815856237s) [upload=12963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:08:48.493962093 (12h28m48.08330708s) [upload=12963]: Done uploading segments... Aug 7 22:08:48.493980028 (12h28m48.083325015s) [upload=12963]: Upload finished. err: <nil> Aug 7 22:09:16.735671384 (12h29m16.325016371s) [upload=12973]: Starting upload Aug 7 22:09:16.735728593 (12h29m16.32507358s) [upload=12973]: Done starting upload Aug 7 22:09:16.735770444 (12h29m16.325115431s) [upload=12973]: Uploading segments... Aug 7 22:09:16.735799409 (12h29m16.325144396s) [upload=12973]: Got next segment. Inline: true Aug 7 22:09:16.735802535 (12h29m16.325147522s) [upload=12973]: Waiting for error group managing segments... Aug 7 22:09:16.735831881 (12h29m16.325176868s) [upload=12973]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:09:16.735835217 (12h29m16.325180204s) [upload=12973]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:09:16.735837421 (12h29m16.325182408s) [upload=12973]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:09:17.021230773 (12h29m16.61057576s) [upload=12973]: Done uploading segments... Aug 7 22:09:17.021247365 (12h29m16.610592352s) [upload=12973]: Upload finished. err: <nil> Aug 7 22:10:03.229004755 (12h30m2.818349752s) [upload=12987]: Starting upload Aug 7 22:10:03.229086592 (12h30m2.818431589s) [upload=12987]: Done starting upload Aug 7 22:10:03.229165603 (12h30m2.8185106s) [upload=12987]: Uploading segments... Aug 7 22:10:03.229268600 (12h30m2.818613618s) [upload=12987]: Got next segment. Inline: true Aug 7 22:10:03.229296584 (12h30m2.818641581s) [upload=12987]: Waiting for error group managing segments... Aug 7 22:10:03.229348544 (12h30m2.818693531s) [upload=12987]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:10:03.229354835 (12h30m2.818699822s) [upload=12987]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:10:03.229357520 (12h30m2.818702517s) [upload=12987]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:03.732013236 (12h30m3.321358233s) [upload=12987]: Done uploading segments... Aug 7 22:10:03.732033424 (12h30m3.321378411s) [upload=12987]: Upload finished. err: <nil> Aug 7 22:11:11.152680180 (12h31m10.742025167s) [upload=13005]: Starting upload Aug 7 22:11:11.152732820 (12h31m10.742077807s) [upload=13005]: Done starting upload Aug 7 22:11:11.152799828 (12h31m10.742144815s) [upload=13005]: Uploading segments... Aug 7 22:11:11.152844534 (12h31m10.742189531s) [upload=13005]: Got next segment. Inline: true Aug 7 22:11:11.152848501 (12h31m10.742193489s) [upload=13005]: Waiting for error group managing segments... Aug 7 22:11:11.152885622 (12h31m10.742230609s) [upload=13005]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:11:11.152889760 (12h31m10.742234747s) [upload=13005]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:11:11.152893338 (12h31m10.742238325s) [upload=13005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:12:10.507162881 (12h32m10.096507878s) [upload=13005]: Done uploading segments... Aug 7 22:12:10.507172128 (12h32m10.096517115s) [upload=13005]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:12:11.394499496 (12h32m10.983844483s) [upload=13024]: Starting upload Aug 7 22:12:11.394565451 (12h32m10.983910429s) [upload=13024]: Done starting upload Aug 7 22:12:11.394614746 (12h32m10.983959733s) [upload=13024]: Uploading segments... Aug 7 22:12:11.394660623 (12h32m10.98400561s) [upload=13024]: Got next segment. Inline: true Aug 7 22:12:11.394666585 (12h32m10.984011572s) [upload=13024]: Waiting for error group managing segments... Aug 7 22:12:11.394716149 (12h32m10.984061136s) [upload=13024]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:11.394722612 (12h32m10.9840676s) [upload=13024]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:12:11.394727622 (12h32m10.984072609s) [upload=13024]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:13:10.756105924 (12h33m10.345450911s) [upload=13024]: Done uploading segments... Aug 7 22:13:10.756126203 (12h33m10.34547118s) [upload=13024]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:13:11.639973323 (12h33m11.22931831s) [upload=13040]: Starting upload Aug 7 22:13:11.640030482 (12h33m11.229375469s) [upload=13040]: Done starting upload Aug 7 22:13:11.640083043 (12h33m11.22942803s) [upload=13040]: Uploading segments... Aug 7 22:13:11.640114102 (12h33m11.229459089s) [upload=13040]: Got next segment. Inline: true Aug 7 22:13:11.640117879 (12h33m11.229462867s) [upload=13040]: Waiting for error group managing segments... Aug 7 22:13:11.640150943 (12h33m11.22949593s) [upload=13040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:13:11.640155241 (12h33m11.229500228s) [upload=13040]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:13:11.640161784 (12h33m11.229506761s) [upload=13040]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:10.987489973 (12h34m10.57683496s) [upload=13040]: Done uploading segments... Aug 7 22:14:10.987508428 (12h34m10.576853415s) [upload=13040]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:14:11.872992660 (12h34m11.462337647s) [upload=13058]: Starting upload Aug 7 22:14:11.873056512 (12h34m11.462401499s) [upload=13058]: Done starting upload Aug 7 22:14:11.873107790 (12h34m11.462452778s) [upload=13058]: Uploading segments... Aug 7 22:14:11.873141936 (12h34m11.462486923s) [upload=13058]: Got next segment. Inline: true Aug 7 22:14:11.873145402 (12h34m11.462490389s) [upload=13058]: Waiting for error group managing segments... Aug 7 22:14:11.873177533 (12h34m11.46252252s) [upload=13058]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:11.873181662 (12h34m11.462526639s) [upload=13058]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:14:11.873183926 (12h34m11.462528913s) [upload=13058]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:15:11.220103017 (12h35m10.809448004s) [upload=13058]: Done uploading segments... Aug 7 22:15:11.220126102 (12h35m10.809471079s) [upload=13058]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:15:12.112939786 (12h35m11.702284773s) [upload=13073]: Starting upload Aug 7 22:15:12.113000451 (12h35m11.702345428s) [upload=13073]: Done starting upload Aug 7 22:15:12.113099632 (12h35m11.702444619s) [upload=13073]: Uploading segments... Aug 7 22:15:12.113176779 (12h35m11.702521766s) [upload=13073]: Got next segment. Inline: true Aug 7 22:15:12.113182540 (12h35m11.702527527s) [upload=13073]: Waiting for error group managing segments... Aug 7 22:15:12.113220422 (12h35m11.702565409s) [upload=13073]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:12.113224931 (12h35m11.702569918s) [upload=13073]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:15:12.113227225 (12h35m11.702572212s) [upload=13073]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:16:11.460295293 (12h36m11.04964029s) [upload=13073]: Done uploading segments... Aug 7 22:16:11.460318257 (12h36m11.049663244s) [upload=13073]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.001024608s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..........X.................X...............................] duration: 1.855244224s FAIL (2025-08-07 22:16:05 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== 8m27.919686751s) [upload=12612]: Done uploading segments... Aug 7 21:48:28.330358527 (12h8m27.919703514s) [upload=12612]: Upload finished. err: <nil> Aug 7 21:48:54.943937458 (12h8m54.533282445s) [upload=12618]: Starting upload Aug 7 21:48:54.944002622 (12h8m54.533347609s) [upload=12618]: Done starting upload Aug 7 21:48:54.944106100 (12h8m54.533451087s) [upload=12618]: Uploading segments... Aug 7 21:48:54.944196242 (12h8m54.53354132s) [upload=12618]: Got next segment. Inline: true Aug 7 21:48:54.944206543 (12h8m54.53355158s) [upload=12618]: Waiting for error group managing segments... Aug 7 21:48:54.944254484 (12h8m54.533599461s) [upload=12618]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:48:54.944259223 (12h8m54.5336042s) [upload=12618]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:48:54.944262139 (12h8m54.533607126s) [upload=12618]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:48:55.330753187 (12h8m54.920098174s) [upload=12618]: Done uploading segments... Aug 7 21:48:55.330774959 (12h8m54.920119936s) [upload=12618]: Upload finished. err: <nil> Aug 7 21:49:36.650951942 (12h9m36.240296929s) [upload=12631]: Starting upload Aug 7 21:49:36.650993252 (12h9m36.240338239s) [upload=12631]: Done starting upload Aug 7 21:49:36.651049029 (12h9m36.240394016s) [upload=12631]: Uploading segments... Aug 7 21:49:36.651075309 (12h9m36.240420286s) [upload=12631]: Got next segment. Inline: true Aug 7 21:49:36.651078775 (12h9m36.240423763s) [upload=12631]: Waiting for error group managing segments... Aug 7 21:49:36.651115776 (12h9m36.240460773s) [upload=12631]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:49:36.651120254 (12h9m36.240465251s) [upload=12631]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:49:36.651124473 (12h9m36.24046947s) [upload=12631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:49:36.920683881 (12h9m36.510028868s) [upload=12631]: Done uploading segments... Aug 7 21:49:36.920707246 (12h9m36.510052233s) [upload=12631]: Upload finished. err: <nil> Aug 7 21:50:03.363429937 (12h10m2.952774924s) [upload=12636]: Starting upload Aug 7 21:50:03.363493408 (12h10m2.952838405s) [upload=12636]: Done starting upload Aug 7 21:50:03.363582518 (12h10m2.952927515s) [upload=12636]: Uploading segments... Aug 7 21:50:03.363692078 (12h10m2.953037085s) [upload=12636]: Got next segment. Inline: true Aug 7 21:50:03.363699752 (12h10m2.953044739s) [upload=12636]: Waiting for error group managing segments... Aug 7 21:50:03.363764116 (12h10m2.953109103s) [upload=12636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:50:03.363773323 (12h10m2.95311831s) [upload=12636]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:50:03.363777271 (12h10m2.953122258s) [upload=12636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:50:03.636022787 (12h10m3.225367774s) [upload=12636]: Done uploading segments... Aug 7 21:50:03.636042204 (12h10m3.225387202s) [upload=12636]: Upload finished. err: <nil> Aug 7 21:50:20.944550175 (12h10m20.533895152s) [upload=12642]: Starting upload Aug 7 21:50:20.944594911 (12h10m20.533939898s) [upload=12642]: Done starting upload Aug 7 21:50:20.944644405 (12h10m20.533989392s) [upload=12642]: Uploading segments... Aug 7 21:50:20.944674473 (12h10m20.534019461s) [upload=12642]: Got next segment. Inline: true Aug 7 21:50:20.944677820 (12h10m20.534022807s) [upload=12642]: Waiting for error group managing segments... Aug 7 21:50:20.944709450 (12h10m20.534054437s) [upload=12642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:50:20.944715131 (12h10m20.534060118s) [upload=12642]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:50:20.944717505 (12h10m20.534062493s) [upload=12642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:50:21.212077476 (12h10m20.801422473s) [upload=12642]: Done uploading segments... Aug 7 21:50:21.212101602 (12h10m20.801446599s) [upload=12642]: Upload finished. err: <nil> Aug 7 21:51:22.649621444 (12h11m22.238966431s) [upload=12660]: Starting upload Aug 7 21:51:22.649696837 (12h11m22.239041824s) [upload=12660]: Done starting upload Aug 7 21:51:22.649777742 (12h11m22.239122729s) [upload=12660]: Uploading segments... Aug 7 21:51:22.649867044 (12h11m22.239212041s) [upload=12660]: Got next segment. Inline: true Aug 7 21:51:22.649872053 (12h11m22.23921704s) [upload=12660]: Waiting for error group managing segments... Aug 7 21:51:22.649940053 (12h11m22.23928504s) [upload=12660]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:51:22.649946785 (12h11m22.239291772s) [upload=12660]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:51:22.649949932 (12h11m22.239294919s) [upload=12660]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:51:22.914527578 (12h11m22.503872565s) [upload=12660]: Done uploading segments... Aug 7 21:51:22.914549130 (12h11m22.503894117s) [upload=12660]: Upload finished. err: <nil> Aug 7 21:51:38.628060070 (12h11m38.217405057s) [upload=12665]: Starting upload Aug 7 21:51:38.628100548 (12h11m38.217445535s) [upload=12665]: Done starting upload Aug 7 21:51:38.628155373 (12h11m38.21750036s) [upload=12665]: Uploading segments... Aug 7 21:51:38.628187814 (12h11m38.217532802s) [upload=12665]: Got next segment. Inline: true Aug 7 21:51:38.628191381 (12h11m38.217536368s) [upload=12665]: Waiting for error group managing segments... Aug 7 21:51:38.628222691 (12h11m38.217567678s) [upload=12665]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:51:38.628226268 (12h11m38.217571255s) [upload=12665]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:51:38.628228662 (12h11m38.217573649s) [upload=12665]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:51:38.940125222 (12h11m38.529470269s) [upload=12665]: Done uploading segments... Aug 7 21:51:38.940147124 (12h11m38.529492191s) [upload=12665]: Upload finished. err: <nil> Aug 7 21:52:16.262879794 (12h12m15.852224781s) [upload=12676]: Starting upload Aug 7 21:52:16.262939168 (12h12m15.852284155s) [upload=12676]: Done starting upload Aug 7 21:52:16.263047876 (12h12m15.852392873s) [upload=12676]: Uploading segments... Aug 7 21:52:16.263139601 (12h12m15.852484598s) [upload=12676]: Got next segment. Inline: true Aug 7 21:52:16.263144440 (12h12m15.852489417s) [upload=12676]: Waiting for error group managing segments... Aug 7 21:52:16.263191931 (12h12m15.852536918s) [upload=12676]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:52:16.263197472 (12h12m15.852542449s) [upload=12676]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:52:16.263200056 (12h12m15.852545043s) [upload=12676]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:52:16.570506954 (12h12m16.159851931s) [upload=12676]: Done uploading segments... Aug 7 21:52:16.570524978 (12h12m16.159869965s) [upload=12676]: Upload finished. err: <nil> Aug 7 21:53:00.574165558 (12h13m0.163510545s) [upload=12691]: Starting upload Aug 7 21:53:00.574212286 (12h13m0.163557273s) [upload=12691]: Done starting upload Aug 7 21:53:00.574248476 (12h13m0.163593463s) [upload=12691]: Uploading segments... Aug 7 21:53:00.574327307 (12h13m0.163672294s) [upload=12691]: Got next segment. Inline: true Aug 7 21:53:00.574332868 (12h13m0.163677855s) [upload=12691]: Waiting for error group managing segments... Aug 7 21:53:00.574373505 (12h13m0.163718492s) [upload=12691]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:53:00.574390598 (12h13m0.163735585s) [upload=12691]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:53:00.574393373 (12h13m0.16373835s) [upload=12691]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:53:00.840335180 (12h13m0.429680167s) [upload=12691]: Done uploading segments... Aug 7 21:53:00.840351500 (12h13m0.429696487s) [upload=12691]: Upload finished. err: <nil> Aug 7 21:53:46.136340579 (12h13m45.725685566s) [upload=12705]: Starting upload Aug 7 21:53:46.136382029 (12h13m45.725727016s) [upload=12705]: Done starting upload Aug 7 21:53:46.136495205 (12h13m45.725840192s) [upload=12705]: Uploading segments... Aug 7 21:53:46.136529601 (12h13m45.725874588s) [upload=12705]: Got next segment. Inline: true Aug 7 21:53:46.136533228 (12h13m45.725878215s) [upload=12705]: Waiting for error group managing segments... Aug 7 21:53:46.136581701 (12h13m45.725926688s) [upload=12705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:53:46.136585949 (12h13m45.725930936s) [upload=12705]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:53:46.136588393 (12h13m45.72593338s) [upload=12705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:53:46.427021148 (12h13m46.016366155s) [upload=12705]: Done uploading segments... Aug 7 21:53:46.427045194 (12h13m46.016390181s) [upload=12705]: Upload finished. err: <nil> Aug 7 21:54:41.670591536 (12h14m41.259936523s) [upload=12721]: Starting upload Aug 7 21:54:41.670632916 (12h14m41.259977903s) [upload=12721]: Done starting upload Aug 7 21:54:41.670677612 (12h14m41.260022599s) [upload=12721]: Uploading segments... Aug 7 21:54:41.670708571 (12h14m41.260053558s) [upload=12721]: Got next segment. Inline: true Aug 7 21:54:41.670713510 (12h14m41.260058497s) [upload=12721]: Waiting for error group managing segments... Aug 7 21:54:41.670746032 (12h14m41.260091019s) [upload=12721]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:54:41.670749979 (12h14m41.260094966s) [upload=12721]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:54:41.670752424 (12h14m41.260097411s) [upload=12721]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:54:41.940759391 (12h14m41.530104368s) [upload=12721]: Done uploading segments... Aug 7 21:54:41.940776734 (12h14m41.530121721s) [upload=12721]: Upload finished. err: <nil> Duration: 1.85526895s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 21 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXX.X...X...................................XXXXX] duration: 30.000520838s FAIL (2025-08-07 22:16:04 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== m43.692372972s) [upload=12960]: Starting upload Aug 7 22:08:44.103093421 (12h28m43.692438408s) [upload=12960]: Done starting upload Aug 7 22:08:44.103142554 (12h28m43.692487541s) [upload=12960]: Uploading segments... Aug 7 22:08:44.103198422 (12h28m43.692543419s) [upload=12960]: Got next segment. Inline: true Aug 7 22:08:44.103202429 (12h28m43.692547416s) [upload=12960]: Waiting for error group managing segments... Aug 7 22:08:44.103235592 (12h28m43.69258058s) [upload=12960]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:08:44.103239219 (12h28m43.692584206s) [upload=12960]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:08:44.103241373 (12h28m43.69258636s) [upload=12960]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:08:44.393696841 (12h28m43.983041818s) [upload=12960]: Done uploading segments... Aug 7 22:08:44.393715166 (12h28m43.983060143s) [upload=12960]: Upload finished. err: <nil> Aug 7 22:09:14.345236113 (12h29m13.9345811s) [upload=12972]: Starting upload Aug 7 22:09:14.345291719 (12h29m13.934636706s) [upload=12972]: Done starting upload Aug 7 22:09:14.345358137 (12h29m13.934703114s) [upload=12972]: Uploading segments... Aug 7 22:09:14.345388645 (12h29m13.934733632s) [upload=12972]: Got next segment. Inline: true Aug 7 22:09:14.345392141 (12h29m13.934737118s) [upload=12972]: Waiting for error group managing segments... Aug 7 22:09:14.345426327 (12h29m13.934771314s) [upload=12972]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:09:14.345430324 (12h29m13.934775301s) [upload=12972]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:09:14.345432839 (12h29m13.934777826s) [upload=12972]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:09:14.834006850 (12h29m14.423351857s) [upload=12972]: Done uploading segments... Aug 7 22:09:14.834026347 (12h29m14.423371334s) [upload=12972]: Upload finished. err: <nil> Aug 7 22:09:53.131289328 (12h29m52.720634315s) [upload=12982]: Starting upload Aug 7 22:09:53.131364442 (12h29m52.720709489s) [upload=12982]: Done starting upload Aug 7 22:09:53.131436210 (12h29m52.720781197s) [upload=12982]: Uploading segments... Aug 7 22:09:53.131465205 (12h29m52.720810192s) [upload=12982]: Got next segment. Inline: true Aug 7 22:09:53.131468902 (12h29m52.720813889s) [upload=12982]: Waiting for error group managing segments... Aug 7 22:09:53.131498809 (12h29m52.720843796s) [upload=12982]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:09:53.131503017 (12h29m52.720848004s) [upload=12982]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:09:53.131505141 (12h29m52.720850118s) [upload=12982]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:09:53.419040148 (12h29m53.008385125s) [upload=12982]: Done uploading segments... Aug 7 22:09:53.419058042 (12h29m53.008403029s) [upload=12982]: Upload finished. err: <nil> Aug 7 22:10:12.329552115 (12h30m11.918897102s) [upload=12989]: Starting upload Aug 7 22:10:12.330067942 (12h30m11.919412929s) [upload=12989]: Done starting upload Aug 7 22:10:12.330139077 (12h30m11.919484054s) [upload=12989]: Uploading segments... Aug 7 22:10:12.330186228 (12h30m11.919531255s) [upload=12989]: Got next segment. Inline: true Aug 7 22:10:12.330192710 (12h30m11.919537737s) [upload=12989]: Waiting for error group managing segments... Aug 7 22:10:12.330240762 (12h30m11.919585749s) [upload=12989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:10:12.330248126 (12h30m11.919593113s) [upload=12989]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:10:12.330252083 (12h30m11.91959709s) [upload=12989]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:12.591107606 (12h30m12.180452593s) [upload=12989]: Done uploading segments... Aug 7 22:10:12.591124920 (12h30m12.180469907s) [upload=12989]: Upload finished. err: <nil> Aug 7 22:10:57.486329607 (12h30m57.075674594s) [upload=13003]: Starting upload Aug 7 22:10:57.486414950 (12h30m57.075759927s) [upload=13003]: Done starting upload Aug 7 22:10:57.486491847 (12h30m57.075836834s) [upload=13003]: Uploading segments... Aug 7 22:10:57.486573694 (12h30m57.075918681s) [upload=13003]: Got next segment. Inline: true Aug 7 22:10:57.486578242 (12h30m57.07592322s) [upload=13003]: Waiting for error group managing segments... Aug 7 22:10:57.486645681 (12h30m57.075990658s) [upload=13003]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:10:57.486650361 (12h30m57.075995348s) [upload=13003]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:10:57.486652795 (12h30m57.075997782s) [upload=13003]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:57.751495816 (12h30m57.340840803s) [upload=13003]: Done uploading segments... Aug 7 22:10:57.751515013 (12h30m57.34086s) [upload=13003]: Upload finished. err: <nil> Aug 7 22:11:44.715949867 (12h31m44.305294854s) [upload=13015]: Starting upload Aug 7 22:11:44.716016213 (12h31m44.3053612s) [upload=13015]: Done starting upload Aug 7 22:11:44.716066600 (12h31m44.305411587s) [upload=13015]: Uploading segments... Aug 7 22:11:44.716104322 (12h31m44.305449309s) [upload=13015]: Got next segment. Inline: true Aug 7 22:11:44.716107879 (12h31m44.305452866s) [upload=13015]: Waiting for error group managing segments... Aug 7 22:11:44.716139289 (12h31m44.305484276s) [upload=13015]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:11:44.716142826 (12h31m44.305487803s) [upload=13015]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:11:44.716146493 (12h31m44.30549147s) [upload=13015]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:12:14.067179909 (12h32m13.656524896s) [upload=13015]: Done uploading segments... Aug 7 22:12:14.067200478 (12h32m13.656545465s) [upload=13015]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:12:53.398168829 (12h32m52.987513816s) [upload=13035]: Starting upload Aug 7 22:12:53.398221350 (12h32m52.987566337s) [upload=13035]: Done starting upload Aug 7 22:12:53.398263170 (12h32m52.987608147s) [upload=13035]: Uploading segments... Aug 7 22:12:53.398291284 (12h32m52.987636261s) [upload=13035]: Got next segment. Inline: true Aug 7 22:12:53.398295111 (12h32m52.987640088s) [upload=13035]: Waiting for error group managing segments... Aug 7 22:12:53.398338094 (12h32m52.987683081s) [upload=13035]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:53.398344366 (12h32m52.987689363s) [upload=13035]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:12:53.398348754 (12h32m52.987693771s) [upload=13035]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:13:22.757670742 (12h33m22.347015729s) [upload=13035]: Done uploading segments... Aug 7 22:13:22.757697282 (12h33m22.34704227s) [upload=13035]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:14:04.336472471 (12h34m3.925817458s) [upload=13057]: Starting upload Aug 7 22:14:04.336528488 (12h34m3.925873475s) [upload=13057]: Done starting upload Aug 7 22:14:04.336571460 (12h34m3.925916447s) [upload=13057]: Uploading segments... Aug 7 22:14:04.336601788 (12h34m3.925946775s) [upload=13057]: Got next segment. Inline: true Aug 7 22:14:04.336605075 (12h34m3.925950052s) [upload=13057]: Waiting for error group managing segments... Aug 7 22:14:04.336639170 (12h34m3.925984147s) [upload=13057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:04.336643388 (12h34m3.925988375s) [upload=13057]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:14:04.336645732 (12h34m3.925990719s) [upload=13057]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:33.693506183 (12h34m33.28285116s) [upload=13057]: Done uploading segments... Aug 7 22:14:33.693533705 (12h34m33.282878692s) [upload=13057]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:14:56.898136222 (12h34m56.487481239s) [upload=13071]: Starting upload Aug 7 22:14:56.898259367 (12h34m56.487604384s) [upload=13071]: Done starting upload Aug 7 22:14:56.898331565 (12h34m56.487676583s) [upload=13071]: Uploading segments... Aug 7 22:14:56.898390859 (12h34m56.487735866s) [upload=13071]: Got next segment. Inline: true Aug 7 22:14:56.898397631 (12h34m56.487742649s) [upload=13071]: Waiting for error group managing segments... Aug 7 22:14:56.898453538 (12h34m56.487798535s) [upload=13071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:56.898460371 (12h34m56.487805388s) [upload=13071]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:14:56.898467315 (12h34m56.487812332s) [upload=13071]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:15:26.250806715 (12h35m25.840151692s) [upload=13071]: Done uploading segments... Aug 7 22:15:26.250836313 (12h35m25.8401813s) [upload=13071]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:15:34.770168401 (12h35m34.359513388s) [upload=13080]: Starting upload Aug 7 22:15:34.770290034 (12h35m34.359635021s) [upload=13080]: Done starting upload Aug 7 22:15:34.770415243 (12h35m34.35976023s) [upload=13080]: Uploading segments... Aug 7 22:15:34.770513081 (12h35m34.359858068s) [upload=13080]: Got next segment. Inline: true Aug 7 22:15:34.770518170 (12h35m34.359863157s) [upload=13080]: Waiting for error group managing segments... Aug 7 22:15:34.770576111 (12h35m34.359921098s) [upload=13080]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:34.770582192 (12h35m34.359927169s) [upload=13080]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:15:34.770584747 (12h35m34.359929734s) [upload=13080]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:16:04.123598264 (12h36m3.712943251s) [upload=13080]: Done uploading segments... Aug 7 22:16:04.123618482 (12h36m3.712963469s) [upload=13080]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.000539814s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXX.X.X.X................................XXXXX] duration: 30.001235239s FAIL (2025-08-07 22:16:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 6TD6257G7TAA, seg_pos={0 0}, upload=13000]: Uploading piece... Aug 7 22:10:53.211637383 (12h30m52.80098237s) [noise=true , address=104.36.82.6:4873 , piece_id=FIZQQ56RDZJUYC2JM24K5UMWTQI64SLS56HH54F4EUIXCWH46FOQ, seg_pos={0 0}, upload=13000]: Uploading piece... Aug 7 22:10:53.211831044 (12h30m52.801176031s) [noise=true , address=45.91.116.13:4154 , piece_id=TPFKH6V3LAM6B2O7L744BZPLT64I2BZVYJXFPEPLIGMVEQ7246YQ, seg_pos={0 0}, upload=13000]: Uploading piece... Aug 7 22:10:53.212051075 (12h30m52.801396062s) [noise=true , address=77.90.31.11:4555 , piece_id=A6WWFITSRKE4GVTQODOYU36ORK7OVE3MVVR7B4FPSSP64DATOQ6A, seg_pos={0 0}, upload=13000]: Uploading piece... Aug 7 22:10:54.951706811 (12h30m54.541051848s) [upload=13000]: Next returned nil segment Aug 7 22:10:54.951730396 (12h30m54.541075413s) [upload=13000]: Waiting for error group managing segments... Aug 7 22:10:55.341563384 (12h30m54.930908361s) [noise=true , address=77.90.31.10:4520 , piece_id=KSMIVSJ4QSOZAXZN7EVTTC4DGFR2TL4JLJQW2NPYYGWAJDKSZ2ZA, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.391397870 (12h30m54.980742857s) [noise=true , address=45.91.116.14:4189 , piece_id=WTRHRDN2HJN3MQI6334ZHVKLMCT7XH7S42MKYCEK5RY2JHDBESOQ, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.429722625 (12h30m55.019067602s) [noise=true , address=77.90.31.12:4590 , piece_id=QCC2XA4NPGVTUZXRRNXV22QVU77S6QB6R4QEQIYUAFWVON3J35RQ, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.430446209 (12h30m55.019791196s) [noise=true , address=104.36.82.6:4873 , piece_id=FIZQQ56RDZJUYC2JM24K5UMWTQI64SLS56HH54F4EUIXCWH46FOQ, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.438319623 (12h30m55.0276646s) [noise=true , address=45.91.116.5:4298 , piece_id=XJIIUMO7PI6XTBLXE7IRPS2YYI53MRGPEW7HS5CWFXUSIXBT5MBQ, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.440502008 (12h30m55.029846995s) [noise=true , address=45.91.116.13:4154 , piece_id=TPFKH6V3LAM6B2O7L744BZPLT64I2BZVYJXFPEPLIGMVEQ7246YQ, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.449485475 (12h30m55.038830462s) [noise=true , address=104.36.82.7:4909 , piece_id=6ZPKL2IEZN54MOXLAC7P4GURLWMJ7MUBNCBXPUGF6TD6257G7TAA, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.449773497 (12h30m55.039118484s) [noise=true , address=77.90.31.13:4627 , piece_id=3IQP6QL756UL7CNL357KLNUTHQCSEG5337GUT2PD2AS623PTIYCA, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.486432234 (12h30m55.075777221s) [noise=true , address=77.90.31.11:4555 , piece_id=A6WWFITSRKE4GVTQODOYU36ORK7OVE3MVVR7B4FPSSP64DATOQ6A, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.522908903 (12h30m55.11225389s) [noise=true , address=77.90.31.18:4805 , piece_id=VQ3CGKCPEKYPCAIOHQ2FZSINJBHBCCQUZD2AD4AIHX72SZN7SEKA, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.546118308 (12h30m55.135463295s) [noise=true , address=104.36.82.4:4802 , piece_id=B4Q65J747GHE2ZRZDM7RLWC3AQC7TLK5D53REAUA5DMHT3TGA44Q, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.568801126 (12h30m55.158146113s) [noise=true , address=104.36.82.4:4804 , piece_id=BI6ZPR56BKHTA2QGM3BNUBFJM6PAD4K27P5TYHRGLG4NBD56LC4A, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.587208293 (12h30m55.176553341s) [noise=true , address=77.90.31.17:4770 , piece_id=BG2ZL7XC52NEJC5FRS4V75MEPJH34B27IPLTJNQLSX4N3OZGVHXA, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.587245686 (12h30m55.176590713s) [seg_pos={0 0}, upload=13000]: Segment reached optimal threshold of 13 pieces. Aug 7 22:10:55.587371516 (12h30m55.176716533s) [noise=true , address=45.91.116.4:4939 , piece_id=FQ4Z6Y4KIOGJLPCZFVPCUJSYOMGDOTZNAIEKGNWKASCLKOWLCTUQ, seg_pos={0 0}, upload=13000]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Aug 7 22:10:55.587442001 (12h30m55.176786988s) [noise=true , address=45.91.116.7:4009 , piece_id=47X2WBQ4XYRNAZDBVU572QSRHXUNSYRCOUALM4SF4WRHCDW6QTCA, seg_pos={0 0}, upload=13000]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Aug 7 22:10:55.587644678 (12h30m55.176989726s) [noise=true , address=104.36.82.12:4150 , piece_id=XG7E3FHDR2VCPREGTWFFJ52JLGUPRUKQJ75KFYEUXPI4OUNTB3ZA, seg_pos={0 0}, upload=13000]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: context canceled; context canceled Aug 7 22:10:55.587681550 (12h30m55.177026557s) [seg_pos={0 0}, upload=13000]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 22:10:55.587777072 (12h30m55.177122109s) [upload=13000]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:10:55.587787912 (12h30m55.177132929s) [upload=13000]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:55.887211865 (12h30m55.476556883s) [upload=13000]: Done uploading segments... Aug 7 22:10:55.887228857 (12h30m55.476573844s) [upload=13000]: Upload finished. err: <nil> Aug 7 22:11:40.872949328 (12h31m40.462294315s) [upload=13014]: Starting upload Aug 7 22:11:40.873003161 (12h31m40.462348148s) [upload=13014]: Done starting upload Aug 7 22:11:40.873072924 (12h31m40.462417911s) [upload=13014]: Uploading segments... Aug 7 22:11:40.873431210 (12h31m40.462776287s) [upload=13014]: Got next segment. Inline: false Aug 7 22:11:40.873471537 (12h31m40.462816525s) [upload=13014]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:11:40.873480314 (12h31m40.462825331s) [upload=13014]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:12:10.229450117 (12h32m9.818795124s) [upload=13014]: Next segment err: context deadline exceeded Aug 7 22:12:10.229589654 (12h32m9.818934631s) [upload=13014]: Done uploading segments... Aug 7 22:12:10.229603972 (12h32m9.818948949s) [upload=13014]: Upload finished. err: context deadline exceeded Aug 7 22:12:48.993762773 (12h32m48.58310776s) [upload=13033]: Starting upload Aug 7 22:12:48.993844269 (12h32m48.583189246s) [upload=13033]: Done starting upload Aug 7 22:12:48.993904404 (12h32m48.583249391s) [upload=13033]: Uploading segments... Aug 7 22:12:48.994022439 (12h32m48.583367427s) [upload=13033]: Got next segment. Inline: false Aug 7 22:12:48.994048309 (12h32m48.583393296s) [upload=13033]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:48.994052647 (12h32m48.583397634s) [upload=13033]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:13:18.347381136 (12h33m17.936726133s) [upload=13033]: Next segment err: context deadline exceeded Aug 7 22:13:18.347514681 (12h33m17.936859668s) [upload=13033]: Done uploading segments... Aug 7 22:13:18.347529670 (12h33m17.936874657s) [upload=13033]: Upload finished. err: context deadline exceeded Aug 7 22:13:51.696413792 (12h33m51.285758769s) [upload=13052]: Starting upload Aug 7 22:13:51.696517580 (12h33m51.285862567s) [upload=13052]: Done starting upload Aug 7 22:13:51.696601842 (12h33m51.285946829s) [upload=13052]: Uploading segments... Aug 7 22:13:51.696787407 (12h33m51.286132414s) [upload=13052]: Got next segment. Inline: false Aug 7 22:13:51.696823415 (12h33m51.286168413s) [upload=13052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:13:51.696829066 (12h33m51.286174073s) [upload=13052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:14:21.042712175 (12h34m20.632057152s) [upload=13052]: Next segment err: context deadline exceeded Aug 7 22:14:21.042793340 (12h34m20.632138327s) [upload=13052]: Done uploading segments... Aug 7 22:14:21.042809000 (12h34m20.632153987s) [upload=13052]: Upload finished. err: context deadline exceeded Aug 7 22:14:36.252577987 (12h34m35.841922974s) [upload=13065]: Starting upload Aug 7 22:14:36.252631760 (12h34m35.841976747s) [upload=13065]: Done starting upload Aug 7 22:14:36.252692656 (12h34m35.842037643s) [upload=13065]: Uploading segments... Aug 7 22:14:36.252805532 (12h34m35.842150519s) [upload=13065]: Got next segment. Inline: false Aug 7 22:14:36.252856870 (12h34m35.842201857s) [upload=13065]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:36.252863202 (12h34m35.842208189s) [upload=13065]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:15:05.607209972 (12h35m5.196554959s) [upload=13065]: Next segment err: context deadline exceeded Aug 7 22:15:05.607234078 (12h35m5.196579065s) [upload=13065]: Done uploading segments... Aug 7 22:15:05.607262002 (12h35m5.196606989s) [upload=13065]: Upload finished. err: context deadline exceeded Aug 7 22:15:30.794674976 (12h35m30.384019963s) [upload=13077]: Starting upload Aug 7 22:15:30.794787000 (12h35m30.384132008s) [upload=13077]: Done starting upload Aug 7 22:15:30.794881030 (12h35m30.384226007s) [upload=13077]: Uploading segments... Aug 7 22:15:30.795138724 (12h35m30.384483711s) [upload=13077]: Got next segment. Inline: false Aug 7 22:15:30.795218847 (12h35m30.384563834s) [upload=13077]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:15:30.795225099 (12h35m30.384570106s) [upload=13077]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:16:00.147601500 (12h35m59.736946487s) [upload=13077]: Next segment err: context deadline exceeded Aug 7 22:16:00.147674640 (12h35m59.737019628s) [upload=13077]: Done uploading segments... Aug 7 22:16:00.147688897 (12h35m59.737033884s) [upload=13077]: Upload finished. err: context deadline exceeded Duration: 30.001332206s core uplink us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 752.461324ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 22:15:48 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 752.498885ms edge gateway vhost us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXX.......X................................XXXXXX] duration: 30.001337817s FAIL (2025-08-07 22:15:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001364708s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......X..X....................X....................X.......] duration: 6.641226818s FAIL (2025-08-07 22:15:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 34e70bda2fe1d7206098dbfb59fc4576ab613f2b752f601da4a5b93473be374e != before ef9ba24509b0003982578e9daec1e6051f8d6ac0bef4f9769b005a8c713045cf ======LOGS====== ding segments... Aug 7 22:10:42.792858699 (12h30m42.382203686s) [upload=12997]: Got next segment. Inline: false Aug 7 22:10:42.792883667 (12h30m42.382228644s) [upload=12997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:10:42.792887915 (12h30m42.382232902s) [upload=12997]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:10:43.239239407 (12h30m42.828584384s) [seg_pos={0 0}, upload=12997]: Begin upload segment... Aug 7 22:10:43.239300844 (12h30m42.828645831s) [seg_pos={0 0}, upload=12997]: Done begin upload segment. Aug 7 22:10:43.239329008 (12h30m42.828673995s) [noise=true , address=104.36.82.61:4394 , piece_id=BZ4OH34HU6COIEBD542MEZ5YGPXPLFMQKIO2EOV23UBTDTFK5XAA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.239541695 (12h30m42.828886692s) [noise=true , address=104.36.82.37:4687 , piece_id=I7ZESSKAS4ZUYX3CUI4ZRZWY2PLRATRNR6FGVGVAXQD6JT3DRUIA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.239923255 (12h30m42.829268242s) [noise=true , address=104.36.82.10:4087 , piece_id=PDZWS5ZZVKC5BCXJIY43NBQVXM7NCLKJZNSVTD3G2K6F2YEY4RVA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.240056509 (12h30m42.829401506s) [noise=true , address=45.91.116.6:4995 , piece_id=GYQIOSTEVRSY6VKX3GQCUC5YZY5S3ZV7R6X2LBG3QRTOOQD53NOA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.240324512 (12h30m42.829669499s) [noise=true , address=104.36.82.38:4705 , piece_id=M557M36RJRMMND4F33G7BOBYVMESQZZN7ZFESJ5CLHEKXWBCL5SA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.240397562 (12h30m42.829742549s) [noise=true , address=79.127.201.215:4351 , piece_id=367MVHSAIIGC6UZQY4EMH25DJCYYTAIUMPFF3PY426RZJHIO44DA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.240579942 (12h30m42.829924929s) [noise=true , address=104.36.82.47:4524 , piece_id=JDRSOKIHNV2RD3COKNJQHRYG2OXOPUBQOG3AHCTYOKKYC3LZISNQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.240814029 (12h30m42.830159016s) [noise=true , address=64.49.13.169:4710 , piece_id=KOUJTCIUNCXFZSJWG6ZKR4KV42WV3ABECSARKOQ5YB5W4LAJTGAA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.240831132 (12h30m42.830176119s) [noise=true , address=104.36.82.59:4331 , piece_id=54OHTXRTR2J4S4HECSB7DFPHSC7T4UGBBC4X7WI72IVCMESI5BQQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.241273939 (12h30m42.830618926s) [noise=true , address=104.36.82.44:4881 , piece_id=YWXSTJSJ46ERQLZ6A2E36CX3R2FI4XVVEZGUMDYJX4ZG3M3R5ACQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.241589863 (12h30m42.83093485s) [noise=true , address=104.36.82.2:4750 , piece_id=24FHN36X2VIWLTYUDHSJNLSOHHEZ56K6DBIGSMI2ZCA26CPT7L7Q, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.241938992 (12h30m42.831283979s) [noise=true , address=104.36.82.3:4791 , piece_id=DJ46BVH37DMTY3YD6VG2AEPNNTPYF7EH4IA3Y633UH3QO6XUTKPQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.242014867 (12h30m42.831359854s) [noise=true , address=104.36.82.58:4270 , piece_id=HLYO6DZ75CNGFXJ6DA5SKQNVPD5COMDHFAETYUZTBOSKMPT4BUEQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.242250207 (12h30m42.831595194s) [noise=true , address=104.36.82.44:4885 , piece_id=FJ726HA3PDRWAYDFGUEK2Y7D3B77ERCFPJSGWNKKPLGWXC3WK7GA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.242484636 (12h30m42.831829623s) [noise=true , address=79.127.205.246:4254 , piece_id=V3AT6KY72VDSIMMOZUC4JECVAQSPAHMAPC5CXDVCO46Z6RTPRUVA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.242691451 (12h30m42.832036438s) [noise=true , address=104.36.82.8:4995 , piece_id=BI77BF2ZGP3ZMB2ABMBUCKQCEF3Z6QHRC7IT6Q2ATIF6VON32WWQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.242894410 (12h30m42.832239397s) [noise=true , address=104.36.82.43:4859 , piece_id=P2S2T5IIUTG4D6EQILFQ7OR5E5JOZHAYO563X7ED5UBN7THKW4QA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.243127415 (12h30m42.832472402s) [noise=true , address=104.36.82.46:4509 , piece_id=WQ3UWQPCMAZ7EDHTSURUQAKL6KWHUA3GSJI6U2G5ZUABRGCTFEXQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:44.351806366 (12h30m43.941151353s) [upload=12997]: Next returned nil segment Aug 7 22:10:44.351819280 (12h30m43.941164257s) [upload=12997]: Waiting for error group managing segments... Aug 7 22:10:44.651888156 (12h30m44.241233144s) [noise=true , address=79.127.205.246:4254 , piece_id=V3AT6KY72VDSIMMOZUC4JECVAQSPAHMAPC5CXDVCO46Z6RTPRUVA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:44.674573650 (12h30m44.263918637s) [noise=true , address=64.49.13.169:4710 , piece_id=KOUJTCIUNCXFZSJWG6ZKR4KV42WV3ABECSARKOQ5YB5W4LAJTGAA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.186610734 (12h30m44.775955721s) [noise=true , address=104.36.82.38:4705 , piece_id=M557M36RJRMMND4F33G7BOBYVMESQZZN7ZFESJ5CLHEKXWBCL5SA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.204653265 (12h30m44.793998252s) [noise=true , address=104.36.82.58:4270 , piece_id=HLYO6DZ75CNGFXJ6DA5SKQNVPD5COMDHFAETYUZTBOSKMPT4BUEQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.218003397 (12h30m44.807348374s) [noise=true , address=79.127.201.215:4351 , piece_id=367MVHSAIIGC6UZQY4EMH25DJCYYTAIUMPFF3PY426RZJHIO44DA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.276599656 (12h30m44.865944643s) [noise=true , address=104.36.82.46:4509 , piece_id=WQ3UWQPCMAZ7EDHTSURUQAKL6KWHUA3GSJI6U2G5ZUABRGCTFEXQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.316254365 (12h30m44.905599352s) [noise=true , address=104.36.82.2:4750 , piece_id=24FHN36X2VIWLTYUDHSJNLSOHHEZ56K6DBIGSMI2ZCA26CPT7L7Q, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.349441480 (12h30m44.938786467s) [noise=true , address=104.36.82.3:4791 , piece_id=DJ46BVH37DMTY3YD6VG2AEPNNTPYF7EH4IA3Y633UH3QO6XUTKPQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.352466728 (12h30m44.941811715s) [noise=true , address=104.36.82.44:4881 , piece_id=YWXSTJSJ46ERQLZ6A2E36CX3R2FI4XVVEZGUMDYJX4ZG3M3R5ACQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.418760245 (12h30m45.008105232s) [noise=true , address=104.36.82.37:4687 , piece_id=I7ZESSKAS4ZUYX3CUI4ZRZWY2PLRATRNR6FGVGVAXQD6JT3DRUIA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.540632266 (12h30m45.129977253s) [noise=true , address=104.36.82.10:4087 , piece_id=PDZWS5ZZVKC5BCXJIY43NBQVXM7NCLKJZNSVTD3G2K6F2YEY4RVA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.567016576 (12h30m45.156361563s) [noise=true , address=104.36.82.59:4331 , piece_id=54OHTXRTR2J4S4HECSB7DFPHSC7T4UGBBC4X7WI72IVCMESI5BQQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.597615539 (12h30m45.186960526s) [noise=true , address=104.36.82.44:4885 , piece_id=FJ726HA3PDRWAYDFGUEK2Y7D3B77ERCFPJSGWNKKPLGWXC3WK7GA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.676238976 (12h30m45.265584013s) [noise=true , address=45.91.116.6:4995 , piece_id=GYQIOSTEVRSY6VKX3GQCUC5YZY5S3ZV7R6X2LBG3QRTOOQD53NOA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.676277709 (12h30m45.265622696s) [seg_pos={0 0}, upload=12997]: Segment reached optimal threshold of 14 pieces. Aug 7 22:10:45.676633781 (12h30m45.265978758s) [noise=true , address=104.36.82.8:4995 , piece_id=BI77BF2ZGP3ZMB2ABMBUCKQCEF3Z6QHRC7IT6Q2ATIF6VON32WWQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tvoTD1LoSj3q2rLiDckfT56dXNKKwdVu1ZAsBZExNFVTom9tH): protocol: expected piece hash; context canceled; EOF Aug 7 22:10:45.676841539 (12h30m45.266186526s) [noise=true , address=104.36.82.43:4859 , piece_id=P2S2T5IIUTG4D6EQILFQ7OR5E5JOZHAYO563X7ED5UBN7THKW4QA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126SqCtRGPziEeopRjA4nNkjkCZWvUwkt5XaMgcqZKJwoxuYWY7): protocol: rpc: dial tcp 104.36.82.43:4859: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:10:45.676869993 (12h30m45.26621498s) [noise=true , address=104.36.82.47:4524 , piece_id=JDRSOKIHNV2RD3COKNJQHRYG2OXOPUBQOG3AHCTYOKKYC3LZISNQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4): protocol: read tcp 188.166.225.170:58322->104.36.82.47:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:10:45.676908757 (12h30m45.266253744s) [noise=true , address=104.36.82.61:4394 , piece_id=BZ4OH34HU6COIEBD542MEZ5YGPXPLFMQKIO2EOV23UBTDTFK5XAA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dSpTqCxeAcL9PVYbP7BhmuKE7papFPc9swB1Fy83WcFDgQbcF): protocol: rpc: dial tcp 104.36.82.61:4394: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:10:45.676927984 (12h30m45.266272961s) [seg_pos={0 0}, upload=12997]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:10:45.677012455 (12h30m45.266357443s) [upload=12997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:10:45.677016543 (12h30m45.26636152s) [upload=12997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:45.998268658 (12h30m45.587613635s) [upload=12997]: Done uploading segments... Aug 7 22:10:45.998285902 (12h30m45.587630889s) [upload=12997]: Upload finished. err: <nil> Duration: 6.641307774s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 19 out of 60 is failed (threshold=3) [XXXXXXXXXXXXX.X........................................XXXXX] duration: 30.000488763s FAIL (2025-08-07 22:15:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000654489s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 21 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXX.X...X...................................XXXX] duration: 30.000960093s FAIL (2025-08-07 22:15:26 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== r: <nil> Aug 7 22:08:06.628735015 (12h28m6.218080002s) [upload=12949]: Starting upload Aug 7 22:08:06.628795140 (12h28m6.218140127s) [upload=12949]: Done starting upload Aug 7 22:08:06.628857911 (12h28m6.218202898s) [upload=12949]: Uploading segments... Aug 7 22:08:06.628887237 (12h28m6.218232224s) [upload=12949]: Got next segment. Inline: true Aug 7 22:08:06.628890713 (12h28m6.2182357s) [upload=12949]: Waiting for error group managing segments... Aug 7 22:08:06.628923866 (12h28m6.218268853s) [upload=12949]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:08:06.628935168 (12h28m6.218280195s) [upload=12949]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:08:06.628938295 (12h28m6.218283272s) [upload=12949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:08:06.909068236 (12h28m6.498413233s) [upload=12949]: Done uploading segments... Aug 7 22:08:06.909091591 (12h28m6.498436598s) [upload=12949]: Upload finished. err: <nil> Aug 7 22:08:44.103027985 (12h28m43.692372972s) [upload=12960]: Starting upload Aug 7 22:08:44.103093421 (12h28m43.692438408s) [upload=12960]: Done starting upload Aug 7 22:08:44.103142554 (12h28m43.692487541s) [upload=12960]: Uploading segments... Aug 7 22:08:44.103198422 (12h28m43.692543419s) [upload=12960]: Got next segment. Inline: true Aug 7 22:08:44.103202429 (12h28m43.692547416s) [upload=12960]: Waiting for error group managing segments... Aug 7 22:08:44.103235592 (12h28m43.69258058s) [upload=12960]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:08:44.103239219 (12h28m43.692584206s) [upload=12960]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:08:44.103241373 (12h28m43.69258636s) [upload=12960]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:08:44.393696841 (12h28m43.983041818s) [upload=12960]: Done uploading segments... Aug 7 22:08:44.393715166 (12h28m43.983060143s) [upload=12960]: Upload finished. err: <nil> Aug 7 22:09:14.345236113 (12h29m13.9345811s) [upload=12972]: Starting upload Aug 7 22:09:14.345291719 (12h29m13.934636706s) [upload=12972]: Done starting upload Aug 7 22:09:14.345358137 (12h29m13.934703114s) [upload=12972]: Uploading segments... Aug 7 22:09:14.345388645 (12h29m13.934733632s) [upload=12972]: Got next segment. Inline: true Aug 7 22:09:14.345392141 (12h29m13.934737118s) [upload=12972]: Waiting for error group managing segments... Aug 7 22:09:14.345426327 (12h29m13.934771314s) [upload=12972]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:09:14.345430324 (12h29m13.934775301s) [upload=12972]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:09:14.345432839 (12h29m13.934777826s) [upload=12972]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:09:14.834006850 (12h29m14.423351857s) [upload=12972]: Done uploading segments... Aug 7 22:09:14.834026347 (12h29m14.423371334s) [upload=12972]: Upload finished. err: <nil> Aug 7 22:09:53.131289328 (12h29m52.720634315s) [upload=12982]: Starting upload Aug 7 22:09:53.131364442 (12h29m52.720709489s) [upload=12982]: Done starting upload Aug 7 22:09:53.131436210 (12h29m52.720781197s) [upload=12982]: Uploading segments... Aug 7 22:09:53.131465205 (12h29m52.720810192s) [upload=12982]: Got next segment. Inline: true Aug 7 22:09:53.131468902 (12h29m52.720813889s) [upload=12982]: Waiting for error group managing segments... Aug 7 22:09:53.131498809 (12h29m52.720843796s) [upload=12982]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:09:53.131503017 (12h29m52.720848004s) [upload=12982]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:09:53.131505141 (12h29m52.720850118s) [upload=12982]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:09:53.419040148 (12h29m53.008385125s) [upload=12982]: Done uploading segments... Aug 7 22:09:53.419058042 (12h29m53.008403029s) [upload=12982]: Upload finished. err: <nil> Aug 7 22:10:12.329552115 (12h30m11.918897102s) [upload=12989]: Starting upload Aug 7 22:10:12.330067942 (12h30m11.919412929s) [upload=12989]: Done starting upload Aug 7 22:10:12.330139077 (12h30m11.919484054s) [upload=12989]: Uploading segments... Aug 7 22:10:12.330186228 (12h30m11.919531255s) [upload=12989]: Got next segment. Inline: true Aug 7 22:10:12.330192710 (12h30m11.919537737s) [upload=12989]: Waiting for error group managing segments... Aug 7 22:10:12.330240762 (12h30m11.919585749s) [upload=12989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:10:12.330248126 (12h30m11.919593113s) [upload=12989]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:10:12.330252083 (12h30m11.91959709s) [upload=12989]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:12.591107606 (12h30m12.180452593s) [upload=12989]: Done uploading segments... Aug 7 22:10:12.591124920 (12h30m12.180469907s) [upload=12989]: Upload finished. err: <nil> Aug 7 22:10:57.486329607 (12h30m57.075674594s) [upload=13003]: Starting upload Aug 7 22:10:57.486414950 (12h30m57.075759927s) [upload=13003]: Done starting upload Aug 7 22:10:57.486491847 (12h30m57.075836834s) [upload=13003]: Uploading segments... Aug 7 22:10:57.486573694 (12h30m57.075918681s) [upload=13003]: Got next segment. Inline: true Aug 7 22:10:57.486578242 (12h30m57.07592322s) [upload=13003]: Waiting for error group managing segments... Aug 7 22:10:57.486645681 (12h30m57.075990658s) [upload=13003]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:10:57.486650361 (12h30m57.075995348s) [upload=13003]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:10:57.486652795 (12h30m57.075997782s) [upload=13003]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:57.751495816 (12h30m57.340840803s) [upload=13003]: Done uploading segments... Aug 7 22:10:57.751515013 (12h30m57.34086s) [upload=13003]: Upload finished. err: <nil> Aug 7 22:11:44.715949867 (12h31m44.305294854s) [upload=13015]: Starting upload Aug 7 22:11:44.716016213 (12h31m44.3053612s) [upload=13015]: Done starting upload Aug 7 22:11:44.716066600 (12h31m44.305411587s) [upload=13015]: Uploading segments... Aug 7 22:11:44.716104322 (12h31m44.305449309s) [upload=13015]: Got next segment. Inline: true Aug 7 22:11:44.716107879 (12h31m44.305452866s) [upload=13015]: Waiting for error group managing segments... Aug 7 22:11:44.716139289 (12h31m44.305484276s) [upload=13015]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:11:44.716142826 (12h31m44.305487803s) [upload=13015]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:11:44.716146493 (12h31m44.30549147s) [upload=13015]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:12:14.067179909 (12h32m13.656524896s) [upload=13015]: Done uploading segments... Aug 7 22:12:14.067200478 (12h32m13.656545465s) [upload=13015]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:12:53.398168829 (12h32m52.987513816s) [upload=13035]: Starting upload Aug 7 22:12:53.398221350 (12h32m52.987566337s) [upload=13035]: Done starting upload Aug 7 22:12:53.398263170 (12h32m52.987608147s) [upload=13035]: Uploading segments... Aug 7 22:12:53.398291284 (12h32m52.987636261s) [upload=13035]: Got next segment. Inline: true Aug 7 22:12:53.398295111 (12h32m52.987640088s) [upload=13035]: Waiting for error group managing segments... Aug 7 22:12:53.398338094 (12h32m52.987683081s) [upload=13035]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:53.398344366 (12h32m52.987689363s) [upload=13035]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:12:53.398348754 (12h32m52.987693771s) [upload=13035]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:13:22.757670742 (12h33m22.347015729s) [upload=13035]: Done uploading segments... Aug 7 22:13:22.757697282 (12h33m22.34704227s) [upload=13035]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:14:04.336472471 (12h34m3.925817458s) [upload=13057]: Starting upload Aug 7 22:14:04.336528488 (12h34m3.925873475s) [upload=13057]: Done starting upload Aug 7 22:14:04.336571460 (12h34m3.925916447s) [upload=13057]: Uploading segments... Aug 7 22:14:04.336601788 (12h34m3.925946775s) [upload=13057]: Got next segment. Inline: true Aug 7 22:14:04.336605075 (12h34m3.925950052s) [upload=13057]: Waiting for error group managing segments... Aug 7 22:14:04.336639170 (12h34m3.925984147s) [upload=13057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:04.336643388 (12h34m3.925988375s) [upload=13057]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:14:04.336645732 (12h34m3.925990719s) [upload=13057]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:33.693506183 (12h34m33.28285116s) [upload=13057]: Done uploading segments... Aug 7 22:14:33.693533705 (12h34m33.282878692s) [upload=13057]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:14:56.898136222 (12h34m56.487481239s) [upload=13071]: Starting upload Aug 7 22:14:56.898259367 (12h34m56.487604384s) [upload=13071]: Done starting upload Aug 7 22:14:56.898331565 (12h34m56.487676583s) [upload=13071]: Uploading segments... Aug 7 22:14:56.898390859 (12h34m56.487735866s) [upload=13071]: Got next segment. Inline: true Aug 7 22:14:56.898397631 (12h34m56.487742649s) [upload=13071]: Waiting for error group managing segments... Aug 7 22:14:56.898453538 (12h34m56.487798535s) [upload=13071]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:56.898460371 (12h34m56.487805388s) [upload=13071]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:14:56.898467315 (12h34m56.487812332s) [upload=13071]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:15:26.250806715 (12h35m25.840151692s) [upload=13071]: Done uploading segments... Aug 7 22:15:26.250836313 (12h35m25.8401813s) [upload=13071]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.000981254s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 25 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXX...X................................XXXXXXX] duration: 30.000502979s FAIL (2025-08-07 22:15:26 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.000522438s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [X..........X.................X..............................] duration: 1.858636253s FAIL (2025-08-07 22:15:24 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== 8m27.919686751s) [upload=12612]: Done uploading segments... Aug 7 21:48:28.330358527 (12h8m27.919703514s) [upload=12612]: Upload finished. err: <nil> Aug 7 21:48:54.943937458 (12h8m54.533282445s) [upload=12618]: Starting upload Aug 7 21:48:54.944002622 (12h8m54.533347609s) [upload=12618]: Done starting upload Aug 7 21:48:54.944106100 (12h8m54.533451087s) [upload=12618]: Uploading segments... Aug 7 21:48:54.944196242 (12h8m54.53354132s) [upload=12618]: Got next segment. Inline: true Aug 7 21:48:54.944206543 (12h8m54.53355158s) [upload=12618]: Waiting for error group managing segments... Aug 7 21:48:54.944254484 (12h8m54.533599461s) [upload=12618]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:48:54.944259223 (12h8m54.5336042s) [upload=12618]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:48:54.944262139 (12h8m54.533607126s) [upload=12618]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:48:55.330753187 (12h8m54.920098174s) [upload=12618]: Done uploading segments... Aug 7 21:48:55.330774959 (12h8m54.920119936s) [upload=12618]: Upload finished. err: <nil> Aug 7 21:49:36.650951942 (12h9m36.240296929s) [upload=12631]: Starting upload Aug 7 21:49:36.650993252 (12h9m36.240338239s) [upload=12631]: Done starting upload Aug 7 21:49:36.651049029 (12h9m36.240394016s) [upload=12631]: Uploading segments... Aug 7 21:49:36.651075309 (12h9m36.240420286s) [upload=12631]: Got next segment. Inline: true Aug 7 21:49:36.651078775 (12h9m36.240423763s) [upload=12631]: Waiting for error group managing segments... Aug 7 21:49:36.651115776 (12h9m36.240460773s) [upload=12631]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:49:36.651120254 (12h9m36.240465251s) [upload=12631]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:49:36.651124473 (12h9m36.24046947s) [upload=12631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:49:36.920683881 (12h9m36.510028868s) [upload=12631]: Done uploading segments... Aug 7 21:49:36.920707246 (12h9m36.510052233s) [upload=12631]: Upload finished. err: <nil> Aug 7 21:50:03.363429937 (12h10m2.952774924s) [upload=12636]: Starting upload Aug 7 21:50:03.363493408 (12h10m2.952838405s) [upload=12636]: Done starting upload Aug 7 21:50:03.363582518 (12h10m2.952927515s) [upload=12636]: Uploading segments... Aug 7 21:50:03.363692078 (12h10m2.953037085s) [upload=12636]: Got next segment. Inline: true Aug 7 21:50:03.363699752 (12h10m2.953044739s) [upload=12636]: Waiting for error group managing segments... Aug 7 21:50:03.363764116 (12h10m2.953109103s) [upload=12636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:50:03.363773323 (12h10m2.95311831s) [upload=12636]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:50:03.363777271 (12h10m2.953122258s) [upload=12636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:50:03.636022787 (12h10m3.225367774s) [upload=12636]: Done uploading segments... Aug 7 21:50:03.636042204 (12h10m3.225387202s) [upload=12636]: Upload finished. err: <nil> Aug 7 21:50:20.944550175 (12h10m20.533895152s) [upload=12642]: Starting upload Aug 7 21:50:20.944594911 (12h10m20.533939898s) [upload=12642]: Done starting upload Aug 7 21:50:20.944644405 (12h10m20.533989392s) [upload=12642]: Uploading segments... Aug 7 21:50:20.944674473 (12h10m20.534019461s) [upload=12642]: Got next segment. Inline: true Aug 7 21:50:20.944677820 (12h10m20.534022807s) [upload=12642]: Waiting for error group managing segments... Aug 7 21:50:20.944709450 (12h10m20.534054437s) [upload=12642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:50:20.944715131 (12h10m20.534060118s) [upload=12642]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:50:20.944717505 (12h10m20.534062493s) [upload=12642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:50:21.212077476 (12h10m20.801422473s) [upload=12642]: Done uploading segments... Aug 7 21:50:21.212101602 (12h10m20.801446599s) [upload=12642]: Upload finished. err: <nil> Aug 7 21:51:22.649621444 (12h11m22.238966431s) [upload=12660]: Starting upload Aug 7 21:51:22.649696837 (12h11m22.239041824s) [upload=12660]: Done starting upload Aug 7 21:51:22.649777742 (12h11m22.239122729s) [upload=12660]: Uploading segments... Aug 7 21:51:22.649867044 (12h11m22.239212041s) [upload=12660]: Got next segment. Inline: true Aug 7 21:51:22.649872053 (12h11m22.23921704s) [upload=12660]: Waiting for error group managing segments... Aug 7 21:51:22.649940053 (12h11m22.23928504s) [upload=12660]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:51:22.649946785 (12h11m22.239291772s) [upload=12660]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:51:22.649949932 (12h11m22.239294919s) [upload=12660]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:51:22.914527578 (12h11m22.503872565s) [upload=12660]: Done uploading segments... Aug 7 21:51:22.914549130 (12h11m22.503894117s) [upload=12660]: Upload finished. err: <nil> Aug 7 21:51:38.628060070 (12h11m38.217405057s) [upload=12665]: Starting upload Aug 7 21:51:38.628100548 (12h11m38.217445535s) [upload=12665]: Done starting upload Aug 7 21:51:38.628155373 (12h11m38.21750036s) [upload=12665]: Uploading segments... Aug 7 21:51:38.628187814 (12h11m38.217532802s) [upload=12665]: Got next segment. Inline: true Aug 7 21:51:38.628191381 (12h11m38.217536368s) [upload=12665]: Waiting for error group managing segments... Aug 7 21:51:38.628222691 (12h11m38.217567678s) [upload=12665]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:51:38.628226268 (12h11m38.217571255s) [upload=12665]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:51:38.628228662 (12h11m38.217573649s) [upload=12665]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:51:38.940125222 (12h11m38.529470269s) [upload=12665]: Done uploading segments... Aug 7 21:51:38.940147124 (12h11m38.529492191s) [upload=12665]: Upload finished. err: <nil> Aug 7 21:52:16.262879794 (12h12m15.852224781s) [upload=12676]: Starting upload Aug 7 21:52:16.262939168 (12h12m15.852284155s) [upload=12676]: Done starting upload Aug 7 21:52:16.263047876 (12h12m15.852392873s) [upload=12676]: Uploading segments... Aug 7 21:52:16.263139601 (12h12m15.852484598s) [upload=12676]: Got next segment. Inline: true Aug 7 21:52:16.263144440 (12h12m15.852489417s) [upload=12676]: Waiting for error group managing segments... Aug 7 21:52:16.263191931 (12h12m15.852536918s) [upload=12676]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:52:16.263197472 (12h12m15.852542449s) [upload=12676]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:52:16.263200056 (12h12m15.852545043s) [upload=12676]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:52:16.570506954 (12h12m16.159851931s) [upload=12676]: Done uploading segments... Aug 7 21:52:16.570524978 (12h12m16.159869965s) [upload=12676]: Upload finished. err: <nil> Aug 7 21:53:00.574165558 (12h13m0.163510545s) [upload=12691]: Starting upload Aug 7 21:53:00.574212286 (12h13m0.163557273s) [upload=12691]: Done starting upload Aug 7 21:53:00.574248476 (12h13m0.163593463s) [upload=12691]: Uploading segments... Aug 7 21:53:00.574327307 (12h13m0.163672294s) [upload=12691]: Got next segment. Inline: true Aug 7 21:53:00.574332868 (12h13m0.163677855s) [upload=12691]: Waiting for error group managing segments... Aug 7 21:53:00.574373505 (12h13m0.163718492s) [upload=12691]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:53:00.574390598 (12h13m0.163735585s) [upload=12691]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:53:00.574393373 (12h13m0.16373835s) [upload=12691]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:53:00.840335180 (12h13m0.429680167s) [upload=12691]: Done uploading segments... Aug 7 21:53:00.840351500 (12h13m0.429696487s) [upload=12691]: Upload finished. err: <nil> Aug 7 21:53:46.136340579 (12h13m45.725685566s) [upload=12705]: Starting upload Aug 7 21:53:46.136382029 (12h13m45.725727016s) [upload=12705]: Done starting upload Aug 7 21:53:46.136495205 (12h13m45.725840192s) [upload=12705]: Uploading segments... Aug 7 21:53:46.136529601 (12h13m45.725874588s) [upload=12705]: Got next segment. Inline: true Aug 7 21:53:46.136533228 (12h13m45.725878215s) [upload=12705]: Waiting for error group managing segments... Aug 7 21:53:46.136581701 (12h13m45.725926688s) [upload=12705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:53:46.136585949 (12h13m45.725930936s) [upload=12705]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:53:46.136588393 (12h13m45.72593338s) [upload=12705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:53:46.427021148 (12h13m46.016366155s) [upload=12705]: Done uploading segments... Aug 7 21:53:46.427045194 (12h13m46.016390181s) [upload=12705]: Upload finished. err: <nil> Aug 7 21:54:41.670591536 (12h14m41.259936523s) [upload=12721]: Starting upload Aug 7 21:54:41.670632916 (12h14m41.259977903s) [upload=12721]: Done starting upload Aug 7 21:54:41.670677612 (12h14m41.260022599s) [upload=12721]: Uploading segments... Aug 7 21:54:41.670708571 (12h14m41.260053558s) [upload=12721]: Got next segment. Inline: true Aug 7 21:54:41.670713510 (12h14m41.260058497s) [upload=12721]: Waiting for error group managing segments... Aug 7 21:54:41.670746032 (12h14m41.260091019s) [upload=12721]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:54:41.670749979 (12h14m41.260094966s) [upload=12721]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:54:41.670752424 (12h14m41.260097411s) [upload=12721]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:54:41.940759391 (12h14m41.530104368s) [upload=12721]: Done uploading segments... Aug 7 21:54:41.940776734 (12h14m41.530121721s) [upload=12721]: Upload finished. err: <nil> Duration: 1.858659729s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 26 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXX.XX..X...............................XXXX] duration: 1m0.001470071s FAIL (2025-08-07 22:15:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== > Aug 7 22:07:24.544076557 (12h27m24.133421534s) [upload=12933]: Starting upload Aug 7 22:07:24.544154457 (12h27m24.133499444s) [upload=12933]: Done starting upload Aug 7 22:07:24.544212037 (12h27m24.133557024s) [upload=12933]: Uploading segments... Aug 7 22:07:24.544273624 (12h27m24.133618611s) [upload=12933]: Got next segment. Inline: true Aug 7 22:07:24.544278614 (12h27m24.133623601s) [upload=12933]: Waiting for error group managing segments... Aug 7 22:07:24.544329832 (12h27m24.133674819s) [upload=12933]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:07:24.544334911 (12h27m24.133679898s) [upload=12933]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:07:24.544337847 (12h27m24.133682834s) [upload=12933]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:07:24.872527048 (12h27m24.461872036s) [upload=12933]: Done uploading segments... Aug 7 22:07:24.872546185 (12h27m24.461891172s) [upload=12933]: Upload finished. err: <nil> Aug 7 22:07:42.678048362 (12h27m42.267393349s) [upload=12938]: Starting upload Aug 7 22:07:42.678122394 (12h27m42.267467381s) [upload=12938]: Done starting upload Aug 7 22:07:42.678184523 (12h27m42.26752951s) [upload=12938]: Uploading segments... Aug 7 22:07:42.678220613 (12h27m42.2675656s) [upload=12938]: Got next segment. Inline: true Aug 7 22:07:42.678225311 (12h27m42.267570299s) [upload=12938]: Waiting for error group managing segments... Aug 7 22:07:42.678261721 (12h27m42.267606708s) [upload=12938]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:07:42.678265758 (12h27m42.267610735s) [upload=12938]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:07:42.678268203 (12h27m42.26761319s) [upload=12938]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:07:42.941204890 (12h27m42.530549877s) [upload=12938]: Done uploading segments... Aug 7 22:07:42.941220821 (12h27m42.530565808s) [upload=12938]: Upload finished. err: <nil> Aug 7 22:08:12.564234560 (12h28m12.153579557s) [upload=12951]: Starting upload Aug 7 22:08:12.564392563 (12h28m12.15373757s) [upload=12951]: Done starting upload Aug 7 22:08:12.564478317 (12h28m12.153823314s) [upload=12951]: Uploading segments... Aug 7 22:08:12.564532671 (12h28m12.153877658s) [upload=12951]: Got next segment. Inline: true Aug 7 22:08:12.564537029 (12h28m12.153882006s) [upload=12951]: Waiting for error group managing segments... Aug 7 22:08:12.564572748 (12h28m12.153917735s) [upload=12951]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:08:12.564577227 (12h28m12.153922204s) [upload=12951]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:08:12.564579511 (12h28m12.153924498s) [upload=12951]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:08:12.853863397 (12h28m12.443208384s) [upload=12951]: Done uploading segments... Aug 7 22:08:12.853882363 (12h28m12.44322735s) [upload=12951]: Upload finished. err: <nil> Aug 7 22:08:48.226332068 (12h28m47.815677055s) [upload=12963]: Starting upload Aug 7 22:08:48.226393765 (12h28m47.815738743s) [upload=12963]: Done starting upload Aug 7 22:08:48.226441998 (12h28m47.815786975s) [upload=12963]: Uploading segments... Aug 7 22:08:48.226471044 (12h28m47.815816031s) [upload=12963]: Got next segment. Inline: true Aug 7 22:08:48.226474330 (12h28m47.815819317s) [upload=12963]: Waiting for error group managing segments... Aug 7 22:08:48.226505239 (12h28m47.815850216s) [upload=12963]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:08:48.226508785 (12h28m47.815853772s) [upload=12963]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:08:48.226511250 (12h28m47.815856237s) [upload=12963]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:08:48.493962093 (12h28m48.08330708s) [upload=12963]: Done uploading segments... Aug 7 22:08:48.493980028 (12h28m48.083325015s) [upload=12963]: Upload finished. err: <nil> Aug 7 22:09:16.735671384 (12h29m16.325016371s) [upload=12973]: Starting upload Aug 7 22:09:16.735728593 (12h29m16.32507358s) [upload=12973]: Done starting upload Aug 7 22:09:16.735770444 (12h29m16.325115431s) [upload=12973]: Uploading segments... Aug 7 22:09:16.735799409 (12h29m16.325144396s) [upload=12973]: Got next segment. Inline: true Aug 7 22:09:16.735802535 (12h29m16.325147522s) [upload=12973]: Waiting for error group managing segments... Aug 7 22:09:16.735831881 (12h29m16.325176868s) [upload=12973]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:09:16.735835217 (12h29m16.325180204s) [upload=12973]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:09:16.735837421 (12h29m16.325182408s) [upload=12973]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:09:17.021230773 (12h29m16.61057576s) [upload=12973]: Done uploading segments... Aug 7 22:09:17.021247365 (12h29m16.610592352s) [upload=12973]: Upload finished. err: <nil> Aug 7 22:10:03.229004755 (12h30m2.818349752s) [upload=12987]: Starting upload Aug 7 22:10:03.229086592 (12h30m2.818431589s) [upload=12987]: Done starting upload Aug 7 22:10:03.229165603 (12h30m2.8185106s) [upload=12987]: Uploading segments... Aug 7 22:10:03.229268600 (12h30m2.818613618s) [upload=12987]: Got next segment. Inline: true Aug 7 22:10:03.229296584 (12h30m2.818641581s) [upload=12987]: Waiting for error group managing segments... Aug 7 22:10:03.229348544 (12h30m2.818693531s) [upload=12987]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:10:03.229354835 (12h30m2.818699822s) [upload=12987]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:10:03.229357520 (12h30m2.818702517s) [upload=12987]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:03.732013236 (12h30m3.321358233s) [upload=12987]: Done uploading segments... Aug 7 22:10:03.732033424 (12h30m3.321378411s) [upload=12987]: Upload finished. err: <nil> Aug 7 22:11:11.152680180 (12h31m10.742025167s) [upload=13005]: Starting upload Aug 7 22:11:11.152732820 (12h31m10.742077807s) [upload=13005]: Done starting upload Aug 7 22:11:11.152799828 (12h31m10.742144815s) [upload=13005]: Uploading segments... Aug 7 22:11:11.152844534 (12h31m10.742189531s) [upload=13005]: Got next segment. Inline: true Aug 7 22:11:11.152848501 (12h31m10.742193489s) [upload=13005]: Waiting for error group managing segments... Aug 7 22:11:11.152885622 (12h31m10.742230609s) [upload=13005]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:11:11.152889760 (12h31m10.742234747s) [upload=13005]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:11:11.152893338 (12h31m10.742238325s) [upload=13005]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:12:10.507162881 (12h32m10.096507878s) [upload=13005]: Done uploading segments... Aug 7 22:12:10.507172128 (12h32m10.096517115s) [upload=13005]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:12:11.394499496 (12h32m10.983844483s) [upload=13024]: Starting upload Aug 7 22:12:11.394565451 (12h32m10.983910429s) [upload=13024]: Done starting upload Aug 7 22:12:11.394614746 (12h32m10.983959733s) [upload=13024]: Uploading segments... Aug 7 22:12:11.394660623 (12h32m10.98400561s) [upload=13024]: Got next segment. Inline: true Aug 7 22:12:11.394666585 (12h32m10.984011572s) [upload=13024]: Waiting for error group managing segments... Aug 7 22:12:11.394716149 (12h32m10.984061136s) [upload=13024]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:11.394722612 (12h32m10.9840676s) [upload=13024]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:12:11.394727622 (12h32m10.984072609s) [upload=13024]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:13:10.756105924 (12h33m10.345450911s) [upload=13024]: Done uploading segments... Aug 7 22:13:10.756126203 (12h33m10.34547118s) [upload=13024]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:13:11.639973323 (12h33m11.22931831s) [upload=13040]: Starting upload Aug 7 22:13:11.640030482 (12h33m11.229375469s) [upload=13040]: Done starting upload Aug 7 22:13:11.640083043 (12h33m11.22942803s) [upload=13040]: Uploading segments... Aug 7 22:13:11.640114102 (12h33m11.229459089s) [upload=13040]: Got next segment. Inline: true Aug 7 22:13:11.640117879 (12h33m11.229462867s) [upload=13040]: Waiting for error group managing segments... Aug 7 22:13:11.640150943 (12h33m11.22949593s) [upload=13040]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:13:11.640155241 (12h33m11.229500228s) [upload=13040]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:13:11.640161784 (12h33m11.229506761s) [upload=13040]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:10.987489973 (12h34m10.57683496s) [upload=13040]: Done uploading segments... Aug 7 22:14:10.987508428 (12h34m10.576853415s) [upload=13040]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:14:11.872992660 (12h34m11.462337647s) [upload=13058]: Starting upload Aug 7 22:14:11.873056512 (12h34m11.462401499s) [upload=13058]: Done starting upload Aug 7 22:14:11.873107790 (12h34m11.462452778s) [upload=13058]: Uploading segments... Aug 7 22:14:11.873141936 (12h34m11.462486923s) [upload=13058]: Got next segment. Inline: true Aug 7 22:14:11.873145402 (12h34m11.462490389s) [upload=13058]: Waiting for error group managing segments... Aug 7 22:14:11.873177533 (12h34m11.46252252s) [upload=13058]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:11.873181662 (12h34m11.462526639s) [upload=13058]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:14:11.873183926 (12h34m11.462528913s) [upload=13058]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:15:11.220103017 (12h35m10.809448004s) [upload=13058]: Done uploading segments... Aug 7 22:15:11.220126102 (12h35m10.809471079s) [upload=13058]: Upload finished. err: metaclient: context deadline exceeded Duration: 1m0.001487444s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXX.......X................................XXXXX] duration: 30.000762374s FAIL (2025-08-07 22:15:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000782323s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 27 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXX...X..X............................XXXX] duration: 1m0.001005241s FAIL (2025-08-07 22:15:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001018366s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXX.X.X.X................................XXXX] duration: 30.000720092s FAIL (2025-08-07 22:15:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 90.31.17:4770 , piece_id=BG2ZL7XC52NEJC5FRS4V75MEPJH34B27IPLTJNQLSX4N3OZGVHXA, seg_pos={0 0}, upload=13000]: Uploading piece... Aug 7 22:10:53.210729946 (12h30m52.800074934s) [noise=true , address=45.91.116.5:4298 , piece_id=XJIIUMO7PI6XTBLXE7IRPS2YYI53MRGPEW7HS5CWFXUSIXBT5MBQ, seg_pos={0 0}, upload=13000]: Uploading piece... Aug 7 22:10:53.210967301 (12h30m52.800312288s) [noise=true , address=45.91.116.14:4189 , piece_id=WTRHRDN2HJN3MQI6334ZHVKLMCT7XH7S42MKYCEK5RY2JHDBESOQ, seg_pos={0 0}, upload=13000]: Uploading piece... Aug 7 22:10:53.211206058 (12h30m52.800551045s) [noise=true , address=104.36.82.12:4150 , piece_id=XG7E3FHDR2VCPREGTWFFJ52JLGUPRUKQJ75KFYEUXPI4OUNTB3ZA, seg_pos={0 0}, upload=13000]: Uploading piece... Aug 7 22:10:53.211450766 (12h30m52.800795753s) [noise=true , address=104.36.82.7:4909 , piece_id=6ZPKL2IEZN54MOXLAC7P4GURLWMJ7MUBNCBXPUGF6TD6257G7TAA, seg_pos={0 0}, upload=13000]: Uploading piece... Aug 7 22:10:53.211637383 (12h30m52.80098237s) [noise=true , address=104.36.82.6:4873 , piece_id=FIZQQ56RDZJUYC2JM24K5UMWTQI64SLS56HH54F4EUIXCWH46FOQ, seg_pos={0 0}, upload=13000]: Uploading piece... Aug 7 22:10:53.211831044 (12h30m52.801176031s) [noise=true , address=45.91.116.13:4154 , piece_id=TPFKH6V3LAM6B2O7L744BZPLT64I2BZVYJXFPEPLIGMVEQ7246YQ, seg_pos={0 0}, upload=13000]: Uploading piece... Aug 7 22:10:53.212051075 (12h30m52.801396062s) [noise=true , address=77.90.31.11:4555 , piece_id=A6WWFITSRKE4GVTQODOYU36ORK7OVE3MVVR7B4FPSSP64DATOQ6A, seg_pos={0 0}, upload=13000]: Uploading piece... Aug 7 22:10:54.951706811 (12h30m54.541051848s) [upload=13000]: Next returned nil segment Aug 7 22:10:54.951730396 (12h30m54.541075413s) [upload=13000]: Waiting for error group managing segments... Aug 7 22:10:55.341563384 (12h30m54.930908361s) [noise=true , address=77.90.31.10:4520 , piece_id=KSMIVSJ4QSOZAXZN7EVTTC4DGFR2TL4JLJQW2NPYYGWAJDKSZ2ZA, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.391397870 (12h30m54.980742857s) [noise=true , address=45.91.116.14:4189 , piece_id=WTRHRDN2HJN3MQI6334ZHVKLMCT7XH7S42MKYCEK5RY2JHDBESOQ, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.429722625 (12h30m55.019067602s) [noise=true , address=77.90.31.12:4590 , piece_id=QCC2XA4NPGVTUZXRRNXV22QVU77S6QB6R4QEQIYUAFWVON3J35RQ, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.430446209 (12h30m55.019791196s) [noise=true , address=104.36.82.6:4873 , piece_id=FIZQQ56RDZJUYC2JM24K5UMWTQI64SLS56HH54F4EUIXCWH46FOQ, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.438319623 (12h30m55.0276646s) [noise=true , address=45.91.116.5:4298 , piece_id=XJIIUMO7PI6XTBLXE7IRPS2YYI53MRGPEW7HS5CWFXUSIXBT5MBQ, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.440502008 (12h30m55.029846995s) [noise=true , address=45.91.116.13:4154 , piece_id=TPFKH6V3LAM6B2O7L744BZPLT64I2BZVYJXFPEPLIGMVEQ7246YQ, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.449485475 (12h30m55.038830462s) [noise=true , address=104.36.82.7:4909 , piece_id=6ZPKL2IEZN54MOXLAC7P4GURLWMJ7MUBNCBXPUGF6TD6257G7TAA, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.449773497 (12h30m55.039118484s) [noise=true , address=77.90.31.13:4627 , piece_id=3IQP6QL756UL7CNL357KLNUTHQCSEG5337GUT2PD2AS623PTIYCA, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.486432234 (12h30m55.075777221s) [noise=true , address=77.90.31.11:4555 , piece_id=A6WWFITSRKE4GVTQODOYU36ORK7OVE3MVVR7B4FPSSP64DATOQ6A, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.522908903 (12h30m55.11225389s) [noise=true , address=77.90.31.18:4805 , piece_id=VQ3CGKCPEKYPCAIOHQ2FZSINJBHBCCQUZD2AD4AIHX72SZN7SEKA, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.546118308 (12h30m55.135463295s) [noise=true , address=104.36.82.4:4802 , piece_id=B4Q65J747GHE2ZRZDM7RLWC3AQC7TLK5D53REAUA5DMHT3TGA44Q, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.568801126 (12h30m55.158146113s) [noise=true , address=104.36.82.4:4804 , piece_id=BI6ZPR56BKHTA2QGM3BNUBFJM6PAD4K27P5TYHRGLG4NBD56LC4A, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.587208293 (12h30m55.176553341s) [noise=true , address=77.90.31.17:4770 , piece_id=BG2ZL7XC52NEJC5FRS4V75MEPJH34B27IPLTJNQLSX4N3OZGVHXA, seg_pos={0 0}, upload=13000]: Done uploading piece. err: <nil> Aug 7 22:10:55.587245686 (12h30m55.176590713s) [seg_pos={0 0}, upload=13000]: Segment reached optimal threshold of 13 pieces. Aug 7 22:10:55.587371516 (12h30m55.176716533s) [noise=true , address=45.91.116.4:4939 , piece_id=FQ4Z6Y4KIOGJLPCZFVPCUJSYOMGDOTZNAIEKGNWKASCLKOWLCTUQ, seg_pos={0 0}, upload=13000]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF Aug 7 22:10:55.587442001 (12h30m55.176786988s) [noise=true , address=45.91.116.7:4009 , piece_id=47X2WBQ4XYRNAZDBVU572QSRHXUNSYRCOUALM4SF4WRHCDW6QTCA, seg_pos={0 0}, upload=13000]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF Aug 7 22:10:55.587644678 (12h30m55.176989726s) [noise=true , address=104.36.82.12:4150 , piece_id=XG7E3FHDR2VCPREGTWFFJ52JLGUPRUKQJ75KFYEUXPI4OUNTB3ZA, seg_pos={0 0}, upload=13000]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK): protocol: context canceled; context canceled Aug 7 22:10:55.587681550 (12h30m55.177026557s) [seg_pos={0 0}, upload=13000]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Aug 7 22:10:55.587777072 (12h30m55.177122109s) [upload=13000]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:10:55.587787912 (12h30m55.177132929s) [upload=13000]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:55.887211865 (12h30m55.476556883s) [upload=13000]: Done uploading segments... Aug 7 22:10:55.887228857 (12h30m55.476573844s) [upload=13000]: Upload finished. err: <nil> Aug 7 22:11:40.872949328 (12h31m40.462294315s) [upload=13014]: Starting upload Aug 7 22:11:40.873003161 (12h31m40.462348148s) [upload=13014]: Done starting upload Aug 7 22:11:40.873072924 (12h31m40.462417911s) [upload=13014]: Uploading segments... Aug 7 22:11:40.873431210 (12h31m40.462776287s) [upload=13014]: Got next segment. Inline: false Aug 7 22:11:40.873471537 (12h31m40.462816525s) [upload=13014]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:11:40.873480314 (12h31m40.462825331s) [upload=13014]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:12:10.229450117 (12h32m9.818795124s) [upload=13014]: Next segment err: context deadline exceeded Aug 7 22:12:10.229589654 (12h32m9.818934631s) [upload=13014]: Done uploading segments... Aug 7 22:12:10.229603972 (12h32m9.818948949s) [upload=13014]: Upload finished. err: context deadline exceeded Aug 7 22:12:48.993762773 (12h32m48.58310776s) [upload=13033]: Starting upload Aug 7 22:12:48.993844269 (12h32m48.583189246s) [upload=13033]: Done starting upload Aug 7 22:12:48.993904404 (12h32m48.583249391s) [upload=13033]: Uploading segments... Aug 7 22:12:48.994022439 (12h32m48.583367427s) [upload=13033]: Got next segment. Inline: false Aug 7 22:12:48.994048309 (12h32m48.583393296s) [upload=13033]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:48.994052647 (12h32m48.583397634s) [upload=13033]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:13:18.347381136 (12h33m17.936726133s) [upload=13033]: Next segment err: context deadline exceeded Aug 7 22:13:18.347514681 (12h33m17.936859668s) [upload=13033]: Done uploading segments... Aug 7 22:13:18.347529670 (12h33m17.936874657s) [upload=13033]: Upload finished. err: context deadline exceeded Aug 7 22:13:51.696413792 (12h33m51.285758769s) [upload=13052]: Starting upload Aug 7 22:13:51.696517580 (12h33m51.285862567s) [upload=13052]: Done starting upload Aug 7 22:13:51.696601842 (12h33m51.285946829s) [upload=13052]: Uploading segments... Aug 7 22:13:51.696787407 (12h33m51.286132414s) [upload=13052]: Got next segment. Inline: false Aug 7 22:13:51.696823415 (12h33m51.286168413s) [upload=13052]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:13:51.696829066 (12h33m51.286174073s) [upload=13052]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:14:21.042712175 (12h34m20.632057152s) [upload=13052]: Next segment err: context deadline exceeded Aug 7 22:14:21.042793340 (12h34m20.632138327s) [upload=13052]: Done uploading segments... Aug 7 22:14:21.042809000 (12h34m20.632153987s) [upload=13052]: Upload finished. err: context deadline exceeded Aug 7 22:14:36.252577987 (12h34m35.841922974s) [upload=13065]: Starting upload Aug 7 22:14:36.252631760 (12h34m35.841976747s) [upload=13065]: Done starting upload Aug 7 22:14:36.252692656 (12h34m35.842037643s) [upload=13065]: Uploading segments... Aug 7 22:14:36.252805532 (12h34m35.842150519s) [upload=13065]: Got next segment. Inline: false Aug 7 22:14:36.252856870 (12h34m35.842201857s) [upload=13065]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:36.252863202 (12h34m35.842208189s) [upload=13065]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:15:05.607209972 (12h35m5.196554959s) [upload=13065]: Next segment err: context deadline exceeded Aug 7 22:15:05.607234078 (12h35m5.196579065s) [upload=13065]: Done uploading segments... Aug 7 22:15:05.607262002 (12h35m5.196606989s) [upload=13065]: Upload finished. err: context deadline exceeded Duration: 30.000740822s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 23 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXX.....X................................XXX] duration: 1m0.00166576s FAIL (2025-08-07 22:15:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: encryption: context deadline exceeded ======LOGS====== 0 0}, upload=13001]: Uploading piece... Aug 7 22:10:54.753681599 (12h30m54.343026586s) [noise=true , address=64.49.13.140:4511 , piece_id=N3RDCDJCVYSZEFI5WGBZGCFBDYYCM3PK4UFM3GACPP7A3ZTWFYBA, seg_pos={0 0}, upload=13001]: Uploading piece... Aug 7 22:10:54.754003535 (12h30m54.343348522s) [noise=true , address=104.36.82.54:4187 , piece_id=5TXRHEQUUJI5OOXJQ4MLO44LQC565ZYA4UJWZ3IQQJAWU7ONTCWQ, seg_pos={0 0}, upload=13001]: Uploading piece... Aug 7 22:10:54.754495266 (12h30m54.343840253s) [noise=true , address=104.36.82.46:4487 , piece_id=567IYX3KLIBE2LXMAGYO52V6MPQ2STZD6NAP5WKKCFI77JC32PXA, seg_pos={0 0}, upload=13001]: Uploading piece... Aug 7 22:10:54.754945627 (12h30m54.344290614s) [noise=true , address=104.36.82.47:4524 , piece_id=NWPFOUA5GSBJ4RG3IXFTTLAECGV6FNKOCGYXVF7ZASXFRHKTWD4A, seg_pos={0 0}, upload=13001]: Uploading piece... Aug 7 22:10:54.755644855 (12h30m54.344989842s) [noise=true , address=77.90.31.15:4704 , piece_id=LSQD2P74HKF4X76B2M257RQNRPU3AVMGC635CWMGKSJMEQ62LAWA, seg_pos={0 0}, upload=13001]: Uploading piece... Aug 7 22:10:54.756057805 (12h30m54.345402792s) [noise=true , address=104.36.82.55:4205 , piece_id=DLG5GJWV7OJ2AB6UMU7Y5ZH5W6BNZGUPT3D375YA4QEHKYLPJXHQ, seg_pos={0 0}, upload=13001]: Uploading piece... Aug 7 22:10:55.874691120 (12h30m55.464036107s) [upload=13001]: Next returned nil segment Aug 7 22:10:55.874712882 (12h30m55.464057869s) [upload=13001]: Waiting for error group managing segments... Aug 7 22:10:56.161312978 (12h30m55.750657966s) [noise=true , address=64.49.13.140:4511 , piece_id=N3RDCDJCVYSZEFI5WGBZGCFBDYYCM3PK4UFM3GACPP7A3ZTWFYBA, seg_pos={0 0}, upload=13001]: Done uploading piece. err: <nil> Aug 7 22:10:56.162308593 (12h30m55.751653581s) [noise=true , address=64.49.13.170:4724 , piece_id=FKWKVJ3OHOPVGP24JDDOATON5JCMJ6NDYHHZMSYKD4QSPGLC3IQQ, seg_pos={0 0}, upload=13001]: Done uploading piece. err: <nil> Aug 7 22:10:56.208707537 (12h30m55.798052524s) [noise=true , address=64.49.13.27:4968 , piece_id=75OO5N6T4S76ADG3JGE6VUEELEBQ3SJI53D2KZS5UQTYBN5JUP5Q, seg_pos={0 0}, upload=13001]: Done uploading piece. err: <nil> Aug 7 22:10:56.289643397 (12h30m55.878988374s) [noise=true , address=193.149.164.143:4702 , piece_id=JUWPBAFHOXTG3ROQ7FM56PXBHDU3IB54I7RRDYN533NKNSNKQMRA, seg_pos={0 0}, upload=13001]: Done uploading piece. err: <nil> Aug 7 22:10:56.717621837 (12h30m56.306966824s) [noise=true , address=104.36.82.55:4205 , piece_id=DLG5GJWV7OJ2AB6UMU7Y5ZH5W6BNZGUPT3D375YA4QEHKYLPJXHQ, seg_pos={0 0}, upload=13001]: Done uploading piece. err: <nil> Aug 7 22:10:56.762634227 (12h30m56.351979214s) [noise=true , address=104.36.82.45:4911 , piece_id=SMAUQHJZL6NGGFEECWS7U2MK53ZHBTFTVZQBNZEUI3GJ7S4DDJ5Q, seg_pos={0 0}, upload=13001]: Done uploading piece. err: <nil> Aug 7 22:10:56.787442963 (12h30m56.37678795s) [noise=true , address=104.36.82.46:4487 , piece_id=567IYX3KLIBE2LXMAGYO52V6MPQ2STZD6NAP5WKKCFI77JC32PXA, seg_pos={0 0}, upload=13001]: Done uploading piece. err: <nil> Aug 7 22:10:56.789302119 (12h30m56.378647106s) [noise=true , address=104.36.82.15:4262 , piece_id=C67QLLAIHDKJQGIWHTB7OBB3XANVOHBTKKKX72VS24VWJPOQ4QHA, seg_pos={0 0}, upload=13001]: Done uploading piece. err: <nil> Aug 7 22:10:56.798785704 (12h30m56.388130691s) [noise=true , address=104.36.82.62:4442 , piece_id=I2R3NEH23OBIAME6WTWRVGWOHDQ3772NE432ZENZUDTHHZKZVJ3A, seg_pos={0 0}, upload=13001]: Done uploading piece. err: <nil> Aug 7 22:10:56.808594450 (12h30m56.397939437s) [noise=true , address=77.90.31.15:4704 , piece_id=LSQD2P74HKF4X76B2M257RQNRPU3AVMGC635CWMGKSJMEQ62LAWA, seg_pos={0 0}, upload=13001]: Done uploading piece. err: <nil> Aug 7 22:10:56.836765378 (12h30m56.426110365s) [noise=true , address=104.36.82.36:4635 , piece_id=IEDOBRNLEVVODT6MBHMTAP4XXZA63XXH4TPWYHHBLDSAD5PXRYBA, seg_pos={0 0}, upload=13001]: Done uploading piece. err: <nil> Aug 7 22:10:56.868605666 (12h30m56.457950653s) [noise=true , address=104.36.82.44:4887 , piece_id=L75OUMWJPYPFMXTL6GUUZYNE6WRUT7FEMCP3NJ53WBUMZ4LV7SZA, seg_pos={0 0}, upload=13001]: Done uploading piece. err: <nil> Aug 7 22:10:56.954035535 (12h30m56.543380522s) [noise=true , address=104.36.82.47:4524 , piece_id=NWPFOUA5GSBJ4RG3IXFTTLAECGV6FNKOCGYXVF7ZASXFRHKTWD4A, seg_pos={0 0}, upload=13001]: Done uploading piece. err: <nil> Aug 7 22:10:57.022668687 (12h30m56.612013664s) [noise=true , address=45.91.116.11:4132 , piece_id=KY43L4P6QWVUQSWCCFYJVXF3Q4TRDBGN6NTJZI6KHROJA673QQTQ, seg_pos={0 0}, upload=13001]: Done uploading piece. err: <nil> Aug 7 22:10:57.022695458 (12h30m56.612040445s) [seg_pos={0 0}, upload=13001]: Segment reached optimal threshold of 14 pieces. Aug 7 22:10:57.022801632 (12h30m56.612146619s) [noise=true , address=104.36.82.60:4340 , piece_id=7WEKMGBEPFZPAAE3J2BHWZYTW6LPSNQQHK7SN4243QXNJGPYAS7A, seg_pos={0 0}, upload=13001]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XeA4A8wbmv7wgPjRQnNbP9xgDQz4MzJH7PasaYsTrsMzTJRhC): protocol: expected piece hash; context canceled; EOF Aug 7 22:10:57.022983821 (12h30m56.612328808s) [noise=true , address=104.36.82.37:4658 , piece_id=UNE7SCJLP6M74UGJAZE6D4QBOEUNO5Q3P3YSS2GOALMHW24BXMTQ, seg_pos={0 0}, upload=13001]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gcSHRechDzTDwivRoAwNFfNj5Zjqvi6oJwVqzbCQinwgWtpPr): protocol: write tcp 188.166.225.170:49650->104.36.82.37:4658: use of closed network connection; write tcp 188.166.225.170:49650->104.36.82.37:4658: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49650->104.36.82.37:4658: use of closed network connection Aug 7 22:10:57.023020220 (12h30m56.612365207s) [noise=true , address=104.36.82.59:4325 , piece_id=FO3UZRA3MKCQ3DDRVDVXFR32LEGJNVVW4HKDIEGKRTBSV53R7E4A, seg_pos={0 0}, upload=13001]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1onAnFbXC1LbUAD31oHGp75v9DpnWmx8LTe6Wazf7XuR3rKnJw): protocol: write tcp 188.166.225.170:53516->104.36.82.59:4325: use of closed network connection; write tcp 188.166.225.170:53516->104.36.82.59:4325: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53516->104.36.82.59:4325: use of closed network connection Aug 7 22:10:57.023052371 (12h30m56.612397358s) [noise=true , address=104.36.82.54:4187 , piece_id=5TXRHEQUUJI5OOXJQ4MLO44LQC565ZYA4UJWZ3IQQJAWU7ONTCWQ, seg_pos={0 0}, upload=13001]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12J7goNH48bnMJyaCAPgUznrX1n6bNFNZXGqpuftNLVLxQxprsF): protocol: read tcp 188.166.225.170:36578->104.36.82.54:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:10:57.023094472 (12h30m56.612439459s) [seg_pos={0 0}, upload=13001]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:10:57.023157663 (12h30m56.61250265s) [upload=13001]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:10:57.023170819 (12h30m56.612515806s) [upload=13001]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:57.312858834 (12h30m56.902203821s) [upload=13001]: Done uploading segments... Aug 7 22:10:57.312875305 (12h30m56.902220292s) [upload=13001]: Upload finished. err: <nil> Aug 7 22:12:03.463576538 (12h32m3.052921515s) [upload=13021]: Starting upload Aug 7 22:12:03.463644107 (12h32m3.052989094s) [upload=13021]: Done starting upload Aug 7 22:12:03.463720814 (12h32m3.053065801s) [upload=13021]: Uploading segments... Aug 7 22:12:03.463803302 (12h32m3.053148289s) [upload=13021]: Got next segment. Inline: false Aug 7 22:12:03.463829302 (12h32m3.053174279s) [upload=13021]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:03.463833710 (12h32m3.053178697s) [upload=13021]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:13:02.821328005 (12h33m2.410672992s) [upload=13021]: Next segment err: context deadline exceeded Aug 7 22:13:02.821359495 (12h33m2.410704482s) [upload=13021]: Done uploading segments... Aug 7 22:13:02.821394342 (12h33m2.410739329s) [upload=13021]: Upload finished. err: context deadline exceeded Aug 7 22:13:03.713207523 (12h33m3.30255251s) [upload=13037]: Starting upload Aug 7 22:13:03.713262448 (12h33m3.302607436s) [upload=13037]: Done starting upload Aug 7 22:13:03.713378641 (12h33m3.302723638s) [upload=13037]: Uploading segments... Aug 7 22:13:03.713520973 (12h33m3.30286596s) [upload=13037]: Got next segment. Inline: false Aug 7 22:13:03.713577020 (12h33m3.302921997s) [upload=13037]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:13:03.713585816 (12h33m3.302930803s) [upload=13037]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:14:03.066118649 (12h34m2.655463636s) [upload=13037]: Next segment err: context deadline exceeded Aug 7 22:14:03.066247905 (12h34m2.655592893s) [upload=13037]: Done uploading segments... Aug 7 22:14:03.066289045 (12h34m2.655634032s) [upload=13037]: Upload finished. err: context deadline exceeded Aug 7 22:14:03.948077576 (12h34m3.537422563s) [upload=13056]: Starting upload Aug 7 22:14:03.948135627 (12h34m3.537480614s) [upload=13056]: Done starting upload Aug 7 22:14:03.948239877 (12h34m3.537584864s) [upload=13056]: Uploading segments... Aug 7 22:14:03.948333165 (12h34m3.537678142s) [upload=13056]: Got next segment. Inline: false Aug 7 22:14:03.948365156 (12h34m3.537710143s) [upload=13056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:03.948369885 (12h34m3.537714872s) [upload=13056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:15:03.303973476 (12h35m2.893318473s) [upload=13056]: Next segment err: context deadline exceeded Aug 7 22:15:03.304134103 (12h35m2.893479091s) [upload=13056]: Done uploading segments... Aug 7 22:15:03.304158811 (12h35m2.893503798s) [upload=13056]: Upload finished. err: context deadline exceeded Duration: 1m0.001680076s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 4 out of 60 is failed (threshold=3) [........X..X....................X....................X......] duration: 8.292498489s FAIL (2025-08-07 22:14:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== digest mismatch when downloading object (sj://uptime-checks-placement-20/orbiter_us_0/4M):: after 34e70bda2fe1d7206098dbfb59fc4576ab613f2b752f601da4a5b93473be374e != before ef9ba24509b0003982578e9daec1e6051f8d6ac0bef4f9769b005a8c713045cf ======LOGS====== ding segments... Aug 7 22:10:42.792858699 (12h30m42.382203686s) [upload=12997]: Got next segment. Inline: false Aug 7 22:10:42.792883667 (12h30m42.382228644s) [upload=12997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:10:42.792887915 (12h30m42.382232902s) [upload=12997]: Flush batch item: pb.BatchRequestItem_SegmentBegin Aug 7 22:10:43.239239407 (12h30m42.828584384s) [seg_pos={0 0}, upload=12997]: Begin upload segment... Aug 7 22:10:43.239300844 (12h30m42.828645831s) [seg_pos={0 0}, upload=12997]: Done begin upload segment. Aug 7 22:10:43.239329008 (12h30m42.828673995s) [noise=true , address=104.36.82.61:4394 , piece_id=BZ4OH34HU6COIEBD542MEZ5YGPXPLFMQKIO2EOV23UBTDTFK5XAA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.239541695 (12h30m42.828886692s) [noise=true , address=104.36.82.37:4687 , piece_id=I7ZESSKAS4ZUYX3CUI4ZRZWY2PLRATRNR6FGVGVAXQD6JT3DRUIA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.239923255 (12h30m42.829268242s) [noise=true , address=104.36.82.10:4087 , piece_id=PDZWS5ZZVKC5BCXJIY43NBQVXM7NCLKJZNSVTD3G2K6F2YEY4RVA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.240056509 (12h30m42.829401506s) [noise=true , address=45.91.116.6:4995 , piece_id=GYQIOSTEVRSY6VKX3GQCUC5YZY5S3ZV7R6X2LBG3QRTOOQD53NOA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.240324512 (12h30m42.829669499s) [noise=true , address=104.36.82.38:4705 , piece_id=M557M36RJRMMND4F33G7BOBYVMESQZZN7ZFESJ5CLHEKXWBCL5SA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.240397562 (12h30m42.829742549s) [noise=true , address=79.127.201.215:4351 , piece_id=367MVHSAIIGC6UZQY4EMH25DJCYYTAIUMPFF3PY426RZJHIO44DA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.240579942 (12h30m42.829924929s) [noise=true , address=104.36.82.47:4524 , piece_id=JDRSOKIHNV2RD3COKNJQHRYG2OXOPUBQOG3AHCTYOKKYC3LZISNQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.240814029 (12h30m42.830159016s) [noise=true , address=64.49.13.169:4710 , piece_id=KOUJTCIUNCXFZSJWG6ZKR4KV42WV3ABECSARKOQ5YB5W4LAJTGAA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.240831132 (12h30m42.830176119s) [noise=true , address=104.36.82.59:4331 , piece_id=54OHTXRTR2J4S4HECSB7DFPHSC7T4UGBBC4X7WI72IVCMESI5BQQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.241273939 (12h30m42.830618926s) [noise=true , address=104.36.82.44:4881 , piece_id=YWXSTJSJ46ERQLZ6A2E36CX3R2FI4XVVEZGUMDYJX4ZG3M3R5ACQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.241589863 (12h30m42.83093485s) [noise=true , address=104.36.82.2:4750 , piece_id=24FHN36X2VIWLTYUDHSJNLSOHHEZ56K6DBIGSMI2ZCA26CPT7L7Q, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.241938992 (12h30m42.831283979s) [noise=true , address=104.36.82.3:4791 , piece_id=DJ46BVH37DMTY3YD6VG2AEPNNTPYF7EH4IA3Y633UH3QO6XUTKPQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.242014867 (12h30m42.831359854s) [noise=true , address=104.36.82.58:4270 , piece_id=HLYO6DZ75CNGFXJ6DA5SKQNVPD5COMDHFAETYUZTBOSKMPT4BUEQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.242250207 (12h30m42.831595194s) [noise=true , address=104.36.82.44:4885 , piece_id=FJ726HA3PDRWAYDFGUEK2Y7D3B77ERCFPJSGWNKKPLGWXC3WK7GA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.242484636 (12h30m42.831829623s) [noise=true , address=79.127.205.246:4254 , piece_id=V3AT6KY72VDSIMMOZUC4JECVAQSPAHMAPC5CXDVCO46Z6RTPRUVA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.242691451 (12h30m42.832036438s) [noise=true , address=104.36.82.8:4995 , piece_id=BI77BF2ZGP3ZMB2ABMBUCKQCEF3Z6QHRC7IT6Q2ATIF6VON32WWQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.242894410 (12h30m42.832239397s) [noise=true , address=104.36.82.43:4859 , piece_id=P2S2T5IIUTG4D6EQILFQ7OR5E5JOZHAYO563X7ED5UBN7THKW4QA, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:43.243127415 (12h30m42.832472402s) [noise=true , address=104.36.82.46:4509 , piece_id=WQ3UWQPCMAZ7EDHTSURUQAKL6KWHUA3GSJI6U2G5ZUABRGCTFEXQ, seg_pos={0 0}, upload=12997]: Uploading piece... Aug 7 22:10:44.351806366 (12h30m43.941151353s) [upload=12997]: Next returned nil segment Aug 7 22:10:44.351819280 (12h30m43.941164257s) [upload=12997]: Waiting for error group managing segments... Aug 7 22:10:44.651888156 (12h30m44.241233144s) [noise=true , address=79.127.205.246:4254 , piece_id=V3AT6KY72VDSIMMOZUC4JECVAQSPAHMAPC5CXDVCO46Z6RTPRUVA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:44.674573650 (12h30m44.263918637s) [noise=true , address=64.49.13.169:4710 , piece_id=KOUJTCIUNCXFZSJWG6ZKR4KV42WV3ABECSARKOQ5YB5W4LAJTGAA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.186610734 (12h30m44.775955721s) [noise=true , address=104.36.82.38:4705 , piece_id=M557M36RJRMMND4F33G7BOBYVMESQZZN7ZFESJ5CLHEKXWBCL5SA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.204653265 (12h30m44.793998252s) [noise=true , address=104.36.82.58:4270 , piece_id=HLYO6DZ75CNGFXJ6DA5SKQNVPD5COMDHFAETYUZTBOSKMPT4BUEQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.218003397 (12h30m44.807348374s) [noise=true , address=79.127.201.215:4351 , piece_id=367MVHSAIIGC6UZQY4EMH25DJCYYTAIUMPFF3PY426RZJHIO44DA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.276599656 (12h30m44.865944643s) [noise=true , address=104.36.82.46:4509 , piece_id=WQ3UWQPCMAZ7EDHTSURUQAKL6KWHUA3GSJI6U2G5ZUABRGCTFEXQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.316254365 (12h30m44.905599352s) [noise=true , address=104.36.82.2:4750 , piece_id=24FHN36X2VIWLTYUDHSJNLSOHHEZ56K6DBIGSMI2ZCA26CPT7L7Q, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.349441480 (12h30m44.938786467s) [noise=true , address=104.36.82.3:4791 , piece_id=DJ46BVH37DMTY3YD6VG2AEPNNTPYF7EH4IA3Y633UH3QO6XUTKPQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.352466728 (12h30m44.941811715s) [noise=true , address=104.36.82.44:4881 , piece_id=YWXSTJSJ46ERQLZ6A2E36CX3R2FI4XVVEZGUMDYJX4ZG3M3R5ACQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.418760245 (12h30m45.008105232s) [noise=true , address=104.36.82.37:4687 , piece_id=I7ZESSKAS4ZUYX3CUI4ZRZWY2PLRATRNR6FGVGVAXQD6JT3DRUIA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.540632266 (12h30m45.129977253s) [noise=true , address=104.36.82.10:4087 , piece_id=PDZWS5ZZVKC5BCXJIY43NBQVXM7NCLKJZNSVTD3G2K6F2YEY4RVA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.567016576 (12h30m45.156361563s) [noise=true , address=104.36.82.59:4331 , piece_id=54OHTXRTR2J4S4HECSB7DFPHSC7T4UGBBC4X7WI72IVCMESI5BQQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.597615539 (12h30m45.186960526s) [noise=true , address=104.36.82.44:4885 , piece_id=FJ726HA3PDRWAYDFGUEK2Y7D3B77ERCFPJSGWNKKPLGWXC3WK7GA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.676238976 (12h30m45.265584013s) [noise=true , address=45.91.116.6:4995 , piece_id=GYQIOSTEVRSY6VKX3GQCUC5YZY5S3ZV7R6X2LBG3QRTOOQD53NOA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: <nil> Aug 7 22:10:45.676277709 (12h30m45.265622696s) [seg_pos={0 0}, upload=12997]: Segment reached optimal threshold of 14 pieces. Aug 7 22:10:45.676633781 (12h30m45.265978758s) [noise=true , address=104.36.82.8:4995 , piece_id=BI77BF2ZGP3ZMB2ABMBUCKQCEF3Z6QHRC7IT6Q2ATIF6VON32WWQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1tvoTD1LoSj3q2rLiDckfT56dXNKKwdVu1ZAsBZExNFVTom9tH): protocol: expected piece hash; context canceled; EOF Aug 7 22:10:45.676841539 (12h30m45.266186526s) [noise=true , address=104.36.82.43:4859 , piece_id=P2S2T5IIUTG4D6EQILFQ7OR5E5JOZHAYO563X7ED5UBN7THKW4QA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:126SqCtRGPziEeopRjA4nNkjkCZWvUwkt5XaMgcqZKJwoxuYWY7): protocol: rpc: dial tcp 104.36.82.43:4859: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:10:45.676869993 (12h30m45.26621498s) [noise=true , address=104.36.82.47:4524 , piece_id=JDRSOKIHNV2RD3COKNJQHRYG2OXOPUBQOG3AHCTYOKKYC3LZISNQ, seg_pos={0 0}, upload=12997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4): protocol: read tcp 188.166.225.170:58322->104.36.82.47:4524: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:10:45.676908757 (12h30m45.266253744s) [noise=true , address=104.36.82.61:4394 , piece_id=BZ4OH34HU6COIEBD542MEZ5YGPXPLFMQKIO2EOV23UBTDTFK5XAA, seg_pos={0 0}, upload=12997]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1dSpTqCxeAcL9PVYbP7BhmuKE7papFPc9swB1Fy83WcFDgQbcF): protocol: rpc: dial tcp 104.36.82.61:4394: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Aug 7 22:10:45.676927984 (12h30m45.266272961s) [seg_pos={0 0}, upload=12997]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Aug 7 22:10:45.677012455 (12h30m45.266357443s) [upload=12997]: Flush batch item: pb.BatchRequestItem_SegmentCommit Aug 7 22:10:45.677016543 (12h30m45.26636152s) [upload=12997]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:45.998268658 (12h30m45.587613635s) [upload=12997]: Done uploading segments... Aug 7 22:10:45.998285902 (12h30m45.587630889s) [upload=12997]: Upload finished. err: <nil> Duration: 8.292525148s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 25 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXX...X................................XXXXXX] duration: 30.001353943s FAIL (2025-08-07 22:14:42 UTC) orbiter warning: operation: Commit uplink: metaclient: context deadline exceeded Duration: 30.001376157s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 3 out of 60 is failed (threshold=3) [.X..........X.................X.............................] duration: 1.86131467s FAIL (2025-08-07 22:14:41 UTC) orbiter warning: operation: ListObjects ======ERROR====== unable to find key in bucket: sj://uptime-checks-placement-20/orbiter_us_0/1K ======LOGS====== 8m27.919686751s) [upload=12612]: Done uploading segments... Aug 7 21:48:28.330358527 (12h8m27.919703514s) [upload=12612]: Upload finished. err: <nil> Aug 7 21:48:54.943937458 (12h8m54.533282445s) [upload=12618]: Starting upload Aug 7 21:48:54.944002622 (12h8m54.533347609s) [upload=12618]: Done starting upload Aug 7 21:48:54.944106100 (12h8m54.533451087s) [upload=12618]: Uploading segments... Aug 7 21:48:54.944196242 (12h8m54.53354132s) [upload=12618]: Got next segment. Inline: true Aug 7 21:48:54.944206543 (12h8m54.53355158s) [upload=12618]: Waiting for error group managing segments... Aug 7 21:48:54.944254484 (12h8m54.533599461s) [upload=12618]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:48:54.944259223 (12h8m54.5336042s) [upload=12618]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:48:54.944262139 (12h8m54.533607126s) [upload=12618]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:48:55.330753187 (12h8m54.920098174s) [upload=12618]: Done uploading segments... Aug 7 21:48:55.330774959 (12h8m54.920119936s) [upload=12618]: Upload finished. err: <nil> Aug 7 21:49:36.650951942 (12h9m36.240296929s) [upload=12631]: Starting upload Aug 7 21:49:36.650993252 (12h9m36.240338239s) [upload=12631]: Done starting upload Aug 7 21:49:36.651049029 (12h9m36.240394016s) [upload=12631]: Uploading segments... Aug 7 21:49:36.651075309 (12h9m36.240420286s) [upload=12631]: Got next segment. Inline: true Aug 7 21:49:36.651078775 (12h9m36.240423763s) [upload=12631]: Waiting for error group managing segments... Aug 7 21:49:36.651115776 (12h9m36.240460773s) [upload=12631]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:49:36.651120254 (12h9m36.240465251s) [upload=12631]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:49:36.651124473 (12h9m36.24046947s) [upload=12631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:49:36.920683881 (12h9m36.510028868s) [upload=12631]: Done uploading segments... Aug 7 21:49:36.920707246 (12h9m36.510052233s) [upload=12631]: Upload finished. err: <nil> Aug 7 21:50:03.363429937 (12h10m2.952774924s) [upload=12636]: Starting upload Aug 7 21:50:03.363493408 (12h10m2.952838405s) [upload=12636]: Done starting upload Aug 7 21:50:03.363582518 (12h10m2.952927515s) [upload=12636]: Uploading segments... Aug 7 21:50:03.363692078 (12h10m2.953037085s) [upload=12636]: Got next segment. Inline: true Aug 7 21:50:03.363699752 (12h10m2.953044739s) [upload=12636]: Waiting for error group managing segments... Aug 7 21:50:03.363764116 (12h10m2.953109103s) [upload=12636]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:50:03.363773323 (12h10m2.95311831s) [upload=12636]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:50:03.363777271 (12h10m2.953122258s) [upload=12636]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:50:03.636022787 (12h10m3.225367774s) [upload=12636]: Done uploading segments... Aug 7 21:50:03.636042204 (12h10m3.225387202s) [upload=12636]: Upload finished. err: <nil> Aug 7 21:50:20.944550175 (12h10m20.533895152s) [upload=12642]: Starting upload Aug 7 21:50:20.944594911 (12h10m20.533939898s) [upload=12642]: Done starting upload Aug 7 21:50:20.944644405 (12h10m20.533989392s) [upload=12642]: Uploading segments... Aug 7 21:50:20.944674473 (12h10m20.534019461s) [upload=12642]: Got next segment. Inline: true Aug 7 21:50:20.944677820 (12h10m20.534022807s) [upload=12642]: Waiting for error group managing segments... Aug 7 21:50:20.944709450 (12h10m20.534054437s) [upload=12642]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:50:20.944715131 (12h10m20.534060118s) [upload=12642]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:50:20.944717505 (12h10m20.534062493s) [upload=12642]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:50:21.212077476 (12h10m20.801422473s) [upload=12642]: Done uploading segments... Aug 7 21:50:21.212101602 (12h10m20.801446599s) [upload=12642]: Upload finished. err: <nil> Aug 7 21:51:22.649621444 (12h11m22.238966431s) [upload=12660]: Starting upload Aug 7 21:51:22.649696837 (12h11m22.239041824s) [upload=12660]: Done starting upload Aug 7 21:51:22.649777742 (12h11m22.239122729s) [upload=12660]: Uploading segments... Aug 7 21:51:22.649867044 (12h11m22.239212041s) [upload=12660]: Got next segment. Inline: true Aug 7 21:51:22.649872053 (12h11m22.23921704s) [upload=12660]: Waiting for error group managing segments... Aug 7 21:51:22.649940053 (12h11m22.23928504s) [upload=12660]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:51:22.649946785 (12h11m22.239291772s) [upload=12660]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:51:22.649949932 (12h11m22.239294919s) [upload=12660]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:51:22.914527578 (12h11m22.503872565s) [upload=12660]: Done uploading segments... Aug 7 21:51:22.914549130 (12h11m22.503894117s) [upload=12660]: Upload finished. err: <nil> Aug 7 21:51:38.628060070 (12h11m38.217405057s) [upload=12665]: Starting upload Aug 7 21:51:38.628100548 (12h11m38.217445535s) [upload=12665]: Done starting upload Aug 7 21:51:38.628155373 (12h11m38.21750036s) [upload=12665]: Uploading segments... Aug 7 21:51:38.628187814 (12h11m38.217532802s) [upload=12665]: Got next segment. Inline: true Aug 7 21:51:38.628191381 (12h11m38.217536368s) [upload=12665]: Waiting for error group managing segments... Aug 7 21:51:38.628222691 (12h11m38.217567678s) [upload=12665]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:51:38.628226268 (12h11m38.217571255s) [upload=12665]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:51:38.628228662 (12h11m38.217573649s) [upload=12665]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:51:38.940125222 (12h11m38.529470269s) [upload=12665]: Done uploading segments... Aug 7 21:51:38.940147124 (12h11m38.529492191s) [upload=12665]: Upload finished. err: <nil> Aug 7 21:52:16.262879794 (12h12m15.852224781s) [upload=12676]: Starting upload Aug 7 21:52:16.262939168 (12h12m15.852284155s) [upload=12676]: Done starting upload Aug 7 21:52:16.263047876 (12h12m15.852392873s) [upload=12676]: Uploading segments... Aug 7 21:52:16.263139601 (12h12m15.852484598s) [upload=12676]: Got next segment. Inline: true Aug 7 21:52:16.263144440 (12h12m15.852489417s) [upload=12676]: Waiting for error group managing segments... Aug 7 21:52:16.263191931 (12h12m15.852536918s) [upload=12676]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:52:16.263197472 (12h12m15.852542449s) [upload=12676]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:52:16.263200056 (12h12m15.852545043s) [upload=12676]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:52:16.570506954 (12h12m16.159851931s) [upload=12676]: Done uploading segments... Aug 7 21:52:16.570524978 (12h12m16.159869965s) [upload=12676]: Upload finished. err: <nil> Aug 7 21:53:00.574165558 (12h13m0.163510545s) [upload=12691]: Starting upload Aug 7 21:53:00.574212286 (12h13m0.163557273s) [upload=12691]: Done starting upload Aug 7 21:53:00.574248476 (12h13m0.163593463s) [upload=12691]: Uploading segments... Aug 7 21:53:00.574327307 (12h13m0.163672294s) [upload=12691]: Got next segment. Inline: true Aug 7 21:53:00.574332868 (12h13m0.163677855s) [upload=12691]: Waiting for error group managing segments... Aug 7 21:53:00.574373505 (12h13m0.163718492s) [upload=12691]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:53:00.574390598 (12h13m0.163735585s) [upload=12691]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:53:00.574393373 (12h13m0.16373835s) [upload=12691]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:53:00.840335180 (12h13m0.429680167s) [upload=12691]: Done uploading segments... Aug 7 21:53:00.840351500 (12h13m0.429696487s) [upload=12691]: Upload finished. err: <nil> Aug 7 21:53:46.136340579 (12h13m45.725685566s) [upload=12705]: Starting upload Aug 7 21:53:46.136382029 (12h13m45.725727016s) [upload=12705]: Done starting upload Aug 7 21:53:46.136495205 (12h13m45.725840192s) [upload=12705]: Uploading segments... Aug 7 21:53:46.136529601 (12h13m45.725874588s) [upload=12705]: Got next segment. Inline: true Aug 7 21:53:46.136533228 (12h13m45.725878215s) [upload=12705]: Waiting for error group managing segments... Aug 7 21:53:46.136581701 (12h13m45.725926688s) [upload=12705]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:53:46.136585949 (12h13m45.725930936s) [upload=12705]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:53:46.136588393 (12h13m45.72593338s) [upload=12705]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:53:46.427021148 (12h13m46.016366155s) [upload=12705]: Done uploading segments... Aug 7 21:53:46.427045194 (12h13m46.016390181s) [upload=12705]: Upload finished. err: <nil> Aug 7 21:54:41.670591536 (12h14m41.259936523s) [upload=12721]: Starting upload Aug 7 21:54:41.670632916 (12h14m41.259977903s) [upload=12721]: Done starting upload Aug 7 21:54:41.670677612 (12h14m41.260022599s) [upload=12721]: Uploading segments... Aug 7 21:54:41.670708571 (12h14m41.260053558s) [upload=12721]: Got next segment. Inline: true Aug 7 21:54:41.670713510 (12h14m41.260058497s) [upload=12721]: Waiting for error group managing segments... Aug 7 21:54:41.670746032 (12h14m41.260091019s) [upload=12721]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 21:54:41.670749979 (12h14m41.260094966s) [upload=12721]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 21:54:41.670752424 (12h14m41.260097411s) [upload=12721]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 21:54:41.940759391 (12h14m41.530104368s) [upload=12721]: Done uploading segments... Aug 7 21:54:41.940776734 (12h14m41.530121721s) [upload=12721]: Upload finished. err: <nil> Duration: 1.861348374s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 19 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXX.X........................................XXXX] duration: 30.000800971s FAIL (2025-08-07 22:14:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.00082086s edge gateway us-select-1
checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway) antiblip: 2 out of 60 is failed (threshold=3) [X...X.......................................................] duration: 746.53585ms remote: "tcp:45.38.24.2:443" target: https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410 FAIL (2025-08-07 22:14:36 UTC) orbiter warning: Get "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410": context deadline exceeded Duration: 746.586667ms edge gateway vhost us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 21 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXX.X...X...................................XXX] duration: 30.001467925s FAIL (2025-08-07 22:14:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== ad=12916]: Upload finished. err: <nil> Aug 7 22:07:18.062211999 (12h27m17.651556986s) [upload=12928]: Starting upload Aug 7 22:07:18.062271193 (12h27m17.651616181s) [upload=12928]: Done starting upload Aug 7 22:07:18.062330397 (12h27m17.651675374s) [upload=12928]: Uploading segments... Aug 7 22:07:18.062360224 (12h27m17.651705211s) [upload=12928]: Got next segment. Inline: true Aug 7 22:07:18.062363690 (12h27m17.651708677s) [upload=12928]: Waiting for error group managing segments... Aug 7 22:07:18.062394940 (12h27m17.651739917s) [upload=12928]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:07:18.062398466 (12h27m17.651743453s) [upload=12928]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:07:18.062400671 (12h27m17.651745648s) [upload=12928]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:07:18.332201201 (12h27m17.921546188s) [upload=12928]: Done uploading segments... Aug 7 22:07:18.332221129 (12h27m17.921566116s) [upload=12928]: Upload finished. err: <nil> Aug 7 22:08:06.628735015 (12h28m6.218080002s) [upload=12949]: Starting upload Aug 7 22:08:06.628795140 (12h28m6.218140127s) [upload=12949]: Done starting upload Aug 7 22:08:06.628857911 (12h28m6.218202898s) [upload=12949]: Uploading segments... Aug 7 22:08:06.628887237 (12h28m6.218232224s) [upload=12949]: Got next segment. Inline: true Aug 7 22:08:06.628890713 (12h28m6.2182357s) [upload=12949]: Waiting for error group managing segments... Aug 7 22:08:06.628923866 (12h28m6.218268853s) [upload=12949]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:08:06.628935168 (12h28m6.218280195s) [upload=12949]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:08:06.628938295 (12h28m6.218283272s) [upload=12949]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:08:06.909068236 (12h28m6.498413233s) [upload=12949]: Done uploading segments... Aug 7 22:08:06.909091591 (12h28m6.498436598s) [upload=12949]: Upload finished. err: <nil> Aug 7 22:08:44.103027985 (12h28m43.692372972s) [upload=12960]: Starting upload Aug 7 22:08:44.103093421 (12h28m43.692438408s) [upload=12960]: Done starting upload Aug 7 22:08:44.103142554 (12h28m43.692487541s) [upload=12960]: Uploading segments... Aug 7 22:08:44.103198422 (12h28m43.692543419s) [upload=12960]: Got next segment. Inline: true Aug 7 22:08:44.103202429 (12h28m43.692547416s) [upload=12960]: Waiting for error group managing segments... Aug 7 22:08:44.103235592 (12h28m43.69258058s) [upload=12960]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:08:44.103239219 (12h28m43.692584206s) [upload=12960]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:08:44.103241373 (12h28m43.69258636s) [upload=12960]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:08:44.393696841 (12h28m43.983041818s) [upload=12960]: Done uploading segments... Aug 7 22:08:44.393715166 (12h28m43.983060143s) [upload=12960]: Upload finished. err: <nil> Aug 7 22:09:14.345236113 (12h29m13.9345811s) [upload=12972]: Starting upload Aug 7 22:09:14.345291719 (12h29m13.934636706s) [upload=12972]: Done starting upload Aug 7 22:09:14.345358137 (12h29m13.934703114s) [upload=12972]: Uploading segments... Aug 7 22:09:14.345388645 (12h29m13.934733632s) [upload=12972]: Got next segment. Inline: true Aug 7 22:09:14.345392141 (12h29m13.934737118s) [upload=12972]: Waiting for error group managing segments... Aug 7 22:09:14.345426327 (12h29m13.934771314s) [upload=12972]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:09:14.345430324 (12h29m13.934775301s) [upload=12972]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:09:14.345432839 (12h29m13.934777826s) [upload=12972]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:09:14.834006850 (12h29m14.423351857s) [upload=12972]: Done uploading segments... Aug 7 22:09:14.834026347 (12h29m14.423371334s) [upload=12972]: Upload finished. err: <nil> Aug 7 22:09:53.131289328 (12h29m52.720634315s) [upload=12982]: Starting upload Aug 7 22:09:53.131364442 (12h29m52.720709489s) [upload=12982]: Done starting upload Aug 7 22:09:53.131436210 (12h29m52.720781197s) [upload=12982]: Uploading segments... Aug 7 22:09:53.131465205 (12h29m52.720810192s) [upload=12982]: Got next segment. Inline: true Aug 7 22:09:53.131468902 (12h29m52.720813889s) [upload=12982]: Waiting for error group managing segments... Aug 7 22:09:53.131498809 (12h29m52.720843796s) [upload=12982]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:09:53.131503017 (12h29m52.720848004s) [upload=12982]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:09:53.131505141 (12h29m52.720850118s) [upload=12982]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:09:53.419040148 (12h29m53.008385125s) [upload=12982]: Done uploading segments... Aug 7 22:09:53.419058042 (12h29m53.008403029s) [upload=12982]: Upload finished. err: <nil> Aug 7 22:10:12.329552115 (12h30m11.918897102s) [upload=12989]: Starting upload Aug 7 22:10:12.330067942 (12h30m11.919412929s) [upload=12989]: Done starting upload Aug 7 22:10:12.330139077 (12h30m11.919484054s) [upload=12989]: Uploading segments... Aug 7 22:10:12.330186228 (12h30m11.919531255s) [upload=12989]: Got next segment. Inline: true Aug 7 22:10:12.330192710 (12h30m11.919537737s) [upload=12989]: Waiting for error group managing segments... Aug 7 22:10:12.330240762 (12h30m11.919585749s) [upload=12989]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:10:12.330248126 (12h30m11.919593113s) [upload=12989]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:10:12.330252083 (12h30m11.91959709s) [upload=12989]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:12.591107606 (12h30m12.180452593s) [upload=12989]: Done uploading segments... Aug 7 22:10:12.591124920 (12h30m12.180469907s) [upload=12989]: Upload finished. err: <nil> Aug 7 22:10:57.486329607 (12h30m57.075674594s) [upload=13003]: Starting upload Aug 7 22:10:57.486414950 (12h30m57.075759927s) [upload=13003]: Done starting upload Aug 7 22:10:57.486491847 (12h30m57.075836834s) [upload=13003]: Uploading segments... Aug 7 22:10:57.486573694 (12h30m57.075918681s) [upload=13003]: Got next segment. Inline: true Aug 7 22:10:57.486578242 (12h30m57.07592322s) [upload=13003]: Waiting for error group managing segments... Aug 7 22:10:57.486645681 (12h30m57.075990658s) [upload=13003]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:10:57.486650361 (12h30m57.075995348s) [upload=13003]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:10:57.486652795 (12h30m57.075997782s) [upload=13003]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:10:57.751495816 (12h30m57.340840803s) [upload=13003]: Done uploading segments... Aug 7 22:10:57.751515013 (12h30m57.34086s) [upload=13003]: Upload finished. err: <nil> Aug 7 22:11:44.715949867 (12h31m44.305294854s) [upload=13015]: Starting upload Aug 7 22:11:44.716016213 (12h31m44.3053612s) [upload=13015]: Done starting upload Aug 7 22:11:44.716066600 (12h31m44.305411587s) [upload=13015]: Uploading segments... Aug 7 22:11:44.716104322 (12h31m44.305449309s) [upload=13015]: Got next segment. Inline: true Aug 7 22:11:44.716107879 (12h31m44.305452866s) [upload=13015]: Waiting for error group managing segments... Aug 7 22:11:44.716139289 (12h31m44.305484276s) [upload=13015]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:11:44.716142826 (12h31m44.305487803s) [upload=13015]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:11:44.716146493 (12h31m44.30549147s) [upload=13015]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:12:14.067179909 (12h32m13.656524896s) [upload=13015]: Done uploading segments... Aug 7 22:12:14.067200478 (12h32m13.656545465s) [upload=13015]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:12:53.398168829 (12h32m52.987513816s) [upload=13035]: Starting upload Aug 7 22:12:53.398221350 (12h32m52.987566337s) [upload=13035]: Done starting upload Aug 7 22:12:53.398263170 (12h32m52.987608147s) [upload=13035]: Uploading segments... Aug 7 22:12:53.398291284 (12h32m52.987636261s) [upload=13035]: Got next segment. Inline: true Aug 7 22:12:53.398295111 (12h32m52.987640088s) [upload=13035]: Waiting for error group managing segments... Aug 7 22:12:53.398338094 (12h32m52.987683081s) [upload=13035]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:12:53.398344366 (12h32m52.987689363s) [upload=13035]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:12:53.398348754 (12h32m52.987693771s) [upload=13035]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:13:22.757670742 (12h33m22.347015729s) [upload=13035]: Done uploading segments... Aug 7 22:13:22.757697282 (12h33m22.34704227s) [upload=13035]: Upload finished. err: metaclient: context deadline exceeded Aug 7 22:14:04.336472471 (12h34m3.925817458s) [upload=13057]: Starting upload Aug 7 22:14:04.336528488 (12h34m3.925873475s) [upload=13057]: Done starting upload Aug 7 22:14:04.336571460 (12h34m3.925916447s) [upload=13057]: Uploading segments... Aug 7 22:14:04.336601788 (12h34m3.925946775s) [upload=13057]: Got next segment. Inline: true Aug 7 22:14:04.336605075 (12h34m3.925950052s) [upload=13057]: Waiting for error group managing segments... Aug 7 22:14:04.336639170 (12h34m3.925984147s) [upload=13057]: Flush batch item: pb.BatchRequestItem_ObjectBegin Aug 7 22:14:04.336643388 (12h34m3.925988375s) [upload=13057]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Aug 7 22:14:04.336645732 (12h34m3.925990719s) [upload=13057]: Flush batch item: pb.BatchRequestItem_ObjectCommit Aug 7 22:14:33.693506183 (12h34m33.28285116s) [upload=13057]: Done uploading segments... Aug 7 22:14:33.693533705 (12h34m33.282878692s) [upload=13057]: Upload finished. err: metaclient: context deadline exceeded Duration: 30.001494106s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.HTTP{"GET" "https://uptime-checks.gateway.storjshare.io/hello-world.txt?AWSAccessKeyId=jw7f74fzbtwblz3mwqolmnivikxq&Signature=azz0x2J96uUqIe6sG77G4wqaf50%3D&Expires=1771549222", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.HTTP{"GET" "https://uptime-checks-us-select.s3.us-select-1.storjapi.io/hello-world.txt?AWSAccessKeyId=jv6fvdi5muos6nw22tdksxgfrd2a&Signature=KjTmfuFyjkIAPapJ1pe5FKXvyIY%3D&Expires=1771562410", ExpectedContent: "Hello, world!"} failure (antiblip window 3/60): (http.us_select_1.gateway)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline)